Method and apparatus for processing image

ABSTRACT

A method of calculating depth information for a three-dimensional (3D) image includes generating a pattern based on the value of at least one cell included in a two-dimensional (2D) image, projecting the pattern, capturing a reflected image of the pattern, and calculating depth information based on the reflected image of the pattern.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2017-0142843, filed on Oct. 30,2017, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a method and apparatus for processing animage, and more particularly, to a method and apparatus for processingtwo-dimensional (2D) and three-dimensional (3D) images.

2. Description of Related Art

As augmented reality (AR) or virtual reality (VR) has attracted moreattention, various electronic devices have been developed to implementAR (or VR).

For AR (or VR) implementation, 3D environment modeling techniques areunder development, including, for example, passive stereoscopy andactive projection.

To model a 3D environment, depth information about an image is needed.Various methods of calculating depth information have been developed,such as structured light-based solutions of projecting a predeterminedcode and calculating depth information on the basis of the projectedcode.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

An aspect of the disclosure is to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea method and apparatus for processing an image for three-dimensional(3D) environment modeling.

In accordance with an aspect of the disclosure, there is provided amethod of calculating depth information. The method includes generatinga pattern based on the value of at least one cell included in a 2Dimage, projecting the pattern, capturing a reflected image of thepattern, and calculating depth information based on the reflected imageof the pattern.

In accordance with another aspect of the disclosure, there is provided aterminal. The terminal includes a projector, a capturing unit, and aprocessor configured to control the projector to generate a patternbased on the value of at least one cell included in a 2D image andproject the pattern, to control the capturing unit to capture areflected image of the pattern, and to calculate depth information basedon the reflected image of the pattern.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses embodiments of the disclosure.

Before undertaking the DETAILED DESCRIPTION below, it may beadvantageous to set forth definitions of certain words and phrases usedthroughout this patent document: the terms “include” and “comprise,” aswell as derivatives thereof, mean inclusion without limitation; the term“or,” is inclusive, meaning and/or; the phrases “associated with” and“associated therewith,” as well as derivatives thereof, may mean toinclude, be included within, interconnect with, contain, be containedwithin, connect to or with, couple to or with, be communicable with,cooperate with, interleave, juxtapose, be proximate to, be bound to orwith, have, have a property of, or the like; and the term “controller”means any device, system or part thereof that controls at least oneoperation, such a device may be implemented in hardware, firmware orsoftware, or some combination of at least two of the same. It should benoted that the functionality associated with any particular controllermay be centralized or distributed, whether locally or remotely.

Moreover, various functions described below can be implemented orsupported by one or more computer programs, each of which is formed fromcomputer readable program code and embodied in a computer readablemedium. The terms “application” and “program” refer to one or morecomputer programs, software components, sets of instructions,procedures, functions, objects, classes, instances, related data, or aportion thereof adapted for implementation in a suitable computerreadable program code. The phrase “computer readable program code”includes any type of computer code, including source code, object code,and executable code. The phrase “computer readable medium” includes anytype of medium capable of being accessed by a computer, such as readonly memory (ROM), random access memory (RAM), a hard disk drive, acompact disc (CD), a digital video disc (DVD), or any other type ofmemory. A “non-transitory” computer readable medium excludes wired,wireless, optical, or other communication links that transporttransitory electrical or other signals. A non-transitory computerreadable medium includes media where data can be permanently stored andmedia where data can be stored and later overwritten, such as arewritahle optical disc or an erasable memory device.

Definitions for certain words and phrases are provided throughout thispatent document. Those of ordinary skill in the art should understandthat in many, if not most instances, such definitions apply to prior, aswell as future uses of such defined words and phrases.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates a three-dimensional (3D) environmental mapreconstruction system according to an embodiment of the disclosure;

FIG. 2 illustrates a terminal according to an embodiment of thedisclosure;

FIG. 3 illustrates a terminal according to another embodiment of thedisclosure;

FIG. 4 illustrates a block diagram of a terminal according to anembodiment of the disclosure;

FIGS. 5A and 5B illustrate neighborhood structures considered fortwo-dimensional (2D) cellular automata according to various embodimentsof the disclosure;

FIG. 6 is a flowchart illustrating a method of generating a differentcellular automaton pattern according to an embodiment of the disclosure;

FIG. 7 illustrates a 2D image according to an embodiment of thedisclosure;

FIG. 8 illustrates a rule for generating a different cellular automatonpattern according to an embodiment of the disclosure;

FIG. 9 illustrates contiguous cells according to an embodiment of thedisclosure;

FIG. 10 illustrates adjacent cells according to an embodiment of thedisclosure;

FIG. 11 is a view depicting a method of setting an initial valueaccording to an embodiment of the disclosure;

FIG. 12 illustrates a result of setting an initial value according to anembodiment of the disclosure:

FIGS. 13-17 illustrate a detailed cellular automation process accordingto an embodiment of the disclosure;

FIG. 18 is a view depicting a method of determining a pitch size for anelectromagnetic field filter, and a cell size for generating a cellularautomaton according to an embodiment of the disclosure;

FIG. 19 illustrates a 2D image according to an embodiment of thedisclosure;

FIG. 20 illustrates a 2D image according to another embodiment of thedisclosure;

FIG. 21 illustrates a process of projecting and capturing a cellularautomaton pattern according to an embodiment of the disclosure;

FIG. 22 illustrates a process of projecting and capturing a cellularautomaton pattern according to another embodiment of the disclosure;

FIG. 23 is a flowchart illustrating a method of decoding a cellularautomaton pattern according to an embodiment of the disclosure;

FIGS. 24A and 24B illustrate a clustering method according to anembodiment of the disclosure;

FIGS. 25A-25E illustrate clustered results according to an embodiment ofthe disclosure;

FIGS. 26A-26F illustrate a method of recognizing a cellular automatonpattern according to an embodiment of the disclosure;

FIG. 27 illustrates indexes in a cellular automaton pattern according toan embodiment of the disclosure;

FIG. 28 illustrates a method of processing indexes in a cellularautomaton pattern according to an embodiment of the disclosure;

FIGS. 29A-29H illustrate a method of detecting horizontal and verticalconnected components according to an embodiment of the disclosure;

FIGS. 30A-30H illustrate a method of estimating a disparity for avertical cluster according to an embodiment of the disclosure;

FIGS. 31A-31D illustrate a method of estimating a disparity for ahorizontal cluster according to an embodiment of the disclosure;

FIGS. 32A and 32B illustrate a method of determining the sum of avertical cluster difference and a horizontal cluster differenceaccording to an embodiment of the disclosure;

FIGS. 33A and 33B illustrate a method of determining a shift tableaccording to an embodiment of the disclosure;

FIGS. 34A-34G illustrate a method of estimating a disparity for avertical cluster according to another embodiment of the disclosure;

FIGS. 35A and 35B illustrate a method of determining a shift tableaccording to another embodiment of the disclosure;

FIGS. 36A-36C illustrate a method of determining a shift table accordingto another embodiment of the disclosure;

FIGS. 37A-37C illustrate a method of determining a shift table accordingto another embodiment of the disclosure;

FIGS. 38A-38E illustrate a method of processing a bordering cellaccording to an embodiment of the disclosure;

FIGS. 39A-39E illustrate a method of processing a bordering cellaccording to another embodiment of the disclosure;

FIGS. 40A-40E illustrate a method of processing a bordering cellaccording to another embodiment of the disclosure;

FIGS. 41A-41E illustrate a method of processing a bordering cellaccording to another embodiment of the disclosure;

FIG. 42 illustrates object segmentation according to an embodiment ofthe disclosure;

FIG. 43 is a flowchart illustrating an object segmentation methodaccording to an embodiment of the disclosure:

FIGS. 44A-44H are views depicting an object segmentation methodaccording to an embodiment of the disclosure;

FIGS. 45A-45F are views depicting an object segmentation methodaccording to another embodiment of the disclosure;

FIGS. 46A-46C illustrate a triangulation method according to anembodiment of the disclosure;

FIG. 47 illustrates the types of cellular automaton grids according tovarious embodiments of the disclosure;

FIG. 48 illustrates an example of cellular automata rules for arectangular grid according to various embodiments of the disclosure;

FIG. 49 illustrates a 2D pattern according to an embodiment of thedisclosure;

FIG. 50 illustrates a 2D pattern according to another embodiment of thedisclosure;

FIG. 51 illustrates cellular automaton evolution according to anembodiment of the disclosure;

FIG. 52 illustrates 2D cellular automata rules according to anembodiment of the disclosure;

FIG. 53 illustrates cellular automata according to an embodiment of thedisclosure;

FIGS. 54A-54D illustrate cellular automata according to anotherembodiment of the disclosure;

FIGS. 55A-55D illustrate self-recovery of a cellular automaton accordingto various embodiments of the disclosure;

FIG. 56 illustrates generation of initial states for a cellularautomaton according to an embodiment of the disclosure;

FIGS. 57A and 57B illustrate patterns generated in an initial stageaccording to an embodiment of the disclosure;

FIG. 58 illustrates a genetic pattern according to various embodimentsof the disclosure;

FIGS. 59A and 59B illustrate matched areas according to variousembodiments of the disclosure:

FIG. 60 illustrates a mapping relationship between a pattern image and acellular automaton grid according to an embodiment of the disclosure;

FIGS. 61A and 61B illustrate mapping between a physical grid and acomputed grid for a cellular automaton grid according to an embodimentof the disclosure;

FIG. 62 illustrates convolution according to an embodiment of thedisclosure;

FIGS. 63A and 63B illustrate the concept of kernel approximationaccording to an embodiment of the disclosure;

FIG. 64 illustrates a convolution computation according to an embodimentof the disclosure;

FIG. 65 illustrates generation of a genetic pattern according to anembodiment of the disclosure;

FIG. 66 illustrates epoch-wise evolution of a 2D cellular automaton, toachieve a quiescent self-organized state according to an embodiment ofthe disclosure;

FIG. 67 illustrates the simplified or approximated performance of a 2Dcellular automaton by evolving the global aspect thereof by means of agenetic algorithm (GA) according to an embodiment of the disclosure;

FIG. 68 illustrates connected components within a transition rule duringapproximation as a convolution kernel according to an embodiment of thedisclosure;

FIG. 69 illustrates reduction of a point cloud density according to anembodiment of the disclosure: and

FIG. 70 is a flowchart illustrating a method of calculating depthinformation for a three-dimensional (3D) image according to anembodiment of the disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

FIGS. 1 through 70, discussed below, and the various embodiments used todescribe the principles of the present disclosure in this patentdocument are by way of illustration only and should not be construed inany way to limit the scope of the disclosure. Those skilled. in the artwill understand that the principles of the present disclosure may beimplemented in any suitably arranged system or device.

Embodiments of the disclosure will be described in detail with referenceto the attached drawings. It is to be appreciated that only parts neededto understand operations according to embodiments of the disclosure willbe described, while the remaining part will not be described lest itshould obscure the subject matter of the disclosure. Although the termsused in the disclosure are defined in consideration of functions in theembodiments of the disclosure, the terms may be changed according to theintention of a user or an operator, or customs. Therefore, thedisclosure should be understood, not simply by the actual terms used butby the meanings of each term lying within.

Many modifications may be made to the disclosure, and the disclosure mayhave various embodiments. Specific embodiments of the disclosure aredescribed with reference to the accompanying drawings. However, theembodiments are not intended to limit the disclosure to the particularembodiments, and it is to be understood that the disclosure coversvarious modifications, equivalents, and/or alternatives to theembodiments within the scope and spirit of the disclosure.

Unless otherwise defined in the embodiments of the disclosure, the termsincluding technical or scientific terms used in the disclosure may havethe same meanings as generally understood by those skilled in the art.The terms as generally defined in dictionaries may be interpreted ashaving the same or similar meanings as or to contextual meanings ofrelated technology. Unless otherwise defined, the terms should not beinterpreted as ideally or excessively formal meanings.

Now, a detailed description will be given of various embodiments of thedisclosure with reference to the attached drawings.

FIG. 1 illustrates a three-dimensional (3D) environmental mapreconstruction system according to an embodiment of the disclosure.

The 3D environmental map reconstruction system may include anelectromagnetic wave (or light) source 22-1, an electromagnetic filteror mask 22, and a light receiving unit 21.

The electromagnetic wave source 22-1 irradiates (or projects) anelectromagnetic wave beam.

The electromagnetic field filter or mask 22 forms a pattern of thespatial distribution of an electromagnetic field to be irradiated ontoan object 29 (or 3D space) on the basis of a lens system.

The light receiving unit 21 detects an electromagnetic field reflectedfrom the object 29 (or 3D space) and records the spatial distribution ofthe electromagnetic field. The light receiving unit 21 may include adetector 21-1.

According to an embodiment of the disclosure, the electromagnetic wavesource 22-1, the electromagnetic field filter or mask 22, and the lightreceiving unit 21 may be included in an electronic device. Theconfiguration of a terminal as an electronic device will be describedbelow in detail.

FIG. 2 illustrates a terminal according to an embodiment of thedisclosure.

Referring to FIG. 2, a terminal 10 includes light projectors (orprojectors) 11 and 12, infrared (IR) cameras 13 and 14, and a red,green, blue (RGB) camera 15.

The light projectors 11 and 12 project a pattern (or a two-dimensional(2D) pattern image).

The IR cameras 13, 14-1, and 14-2 capture an image (e.g., 2D image) ofthe projected pattern reflected from an object.

The RGB camera 15 captures a space or an object, for example, in avisible ray area.

In FIG. 2, the light projectors 11 and 12, and the IR cameras 13, 14-1and 14-2 may be provided as indicated by reference numerals 12 and 13(implementation method 1) or as indicated by reference numerals 11 and14 (implementation 2). For example, the distances b (base) between thelight projectors 11 and 12 and the IR cameras 13, 14-1, and 14-2 may beset to be equal to or less than 100 mm, as indicated by referencenumerals 16 and 18. Further, the RGB camera 15 may be set apart from theIR cameras 14-1 and 14-2 by a distance of 10 mm or larger, as indicatedby reference numeral 17.

FIG. 3 illustrates a terminal according to another embodiment of thedisclosure.

FIG. 3 illustrates the front surface 30-1 and rear surface 30-2 of aterminal 30.

The front surface 30-1 of the terminal 30 includes an IR camera 31 and alight projector 32. The rear surface 30-2 of the terminal 30 includes anIR camera 33, a light projector 34, and an RGB camera 35. A descriptionredundant with the description of FIG. 1 is avoided herein.

FIG. 4 illustrates a block diagram of a terminal according to anembodiment of the disclosure.

A description redundant with the descriptions of FIGS. 1 and 2 isavoided herein.

A terminal 400 includes a projector 410, a capturing unit 420, and aprocessor 430.

The projector 410 projects light. For example, the projector 410 mayproject a predetermined light pattern. Herein, the pattern may beprojected by IR light.

The capturing unit 420 may capture an image of the projected light. Forexample, the capturing unit 420 may capture an image of the projectedlight which has been reflected. The capturing unit 420 may have an RGBcapturing function, or the terminal 400 may further include a capturingunit that has the RGB capturing function.

The processor 430 provides overall control to the terminal 400.Particularly, the processor 430 may generate a pattern by evolving acellular automaton. That is, the processor 430 may generate a pattern onthe basis of the value of at least one cell included in a 2D image. Theprocess of generating a pattern amounts to setting a value for each of aplurality of cells included in the 2D image.

One of at least two values may be assigned to each of the plurality ofcells included in the 2D image. A cell is a unit that forms a 2D imageand a pattern. For example, the at least two values are ‘0’ and ‘1’, and‘0 or ‘1’ may be assigned to each of the plurality of cells. Further,automaton evolution refers to a process of assigning one of at least twovalues to each of a plurality of cells. For example, the processor 430may repeat an operation of assigning ‘0’ or ‘1’ to one of the pluralityof cells and assigning ‘0’ or ‘1’ to one cell neighboring to the cellamong the plurality of cells.

For example, assuming one-dimensional (1D) automaton evolution, theprocessor 430 may perform 1D evolution by the following [Equation 1].

a _(i) ^(t+1)=δ(a _(i−1) ^(t) ,a _(i) ^(t) ,a _(i+1) ^(t))=(a _(i−1)^(t) +a _(i) ^(t) +a _(i+1) ^(t))   [Equation 1]

In another example, assuming 2D automaton evolution, the processor 430may perform 2D evolution by the following [Equation 2].

a _(ij) ^(t+1)=δ(a _(i−1,j) ^(t) ,a _(i,j) ^(t) ,a _(i+1,j) ^(t) ,a_(i,j−1) ^(t) ,a _(i,j+1) ^(t))   [Equation 2]

When generating a pattern, the processor 430 may use an initial valueset in a row or column of a plurality of cells included in a 2D image.For example, if the row or column includes five cells, the initial valueof the row or column may be set to 10010.

In the above example, 1D evolution may be used for 2D evolution.Further, it is apparent that various embodiments of the disclosure areapplicable to 2D or higher-order evolution.

The processor 430 may generate a pattern by determining the value ofanother cell neighboring one cell on the basis of the value of the onecell and the sum of the values of two cells neighboring the one cell inthe 2D image.

The processor 430 may control the projector 410 to project the generatedpattern. Further, the processor 430 may control the capturing unit 420to capture a reflected image of the pattern. Herein, the processor 430may calculate depth information for 3D environment modeling on the basisof the pattern and the reflected image of the pattern.

The processor 430 may generate a pattern on the basis of the values ofcells in a row or column of the plurality of cells included in thereflected image of the pattern. Further, the processor 430 may determinethe value of another cell neighboring a cell on the basis of the valueof the cell and the values of two cells neighboring the cell. Further,the processor 430 may compare the values of the plurality of cellsincluded in the reflected image of the pattern with the values of theplurality of cells included in the pattern generated on the basis of thevalues of the cells in the row or column, and may calculate depthinformation on the basis of the comparison result.

Depth information may be calculated on the basis of a match or mismatchbetween the values of the plurality of cells included in the pattern andthe values of the plurality of cells included in the reflected image ofthe pattern.

The processor 430 may determine a lost area in the reflected image ofthe pattern. In this case, the processor 430 may generate a pattern forthe lost area by automaton evolution.

The processor 430 may determine whether to perform automaton evolutionaccording to whether a first row/column generated on the basis of onerow/column included in the reflected image of the pattern matches asecond row/column neighboring the one row/column. The match or mismatchmay be determined on the basis of the Hamming distance between the firstrow/column and the second row/column.

Hereinbelow, theories and mathematical formulas related to thedisclosure and specific embodiments of the disclosure will be described.Apparently, each theory, mathematic formula, and specific embodiment maybe performed by a processor, even though it is not specified explicitly.

The afore-mentioned pattern may be referred to as a genetic pattern.This genetic pattern may be generated by cellular automata or cellularautomaton evolution.

A method of generating a structured light pattern is provided accordingto an embodiment of the disclosure. A pattern generation process is theevolution of a cellular automaton in a 2D grid with the same resolutionas that of an IR camera image. Cellular automata may be defined as aprocess of determining the values of neighbor cells on the basis of theinitial value of an initial cell (or point) or the initial values ofcells in an initial row or column in an image pattern. Herein, aninitial value may be preset or received externally.

The result of the cellular automaton pattern evolution may be a constant2D pattern based on local and global correlations, and a sub-pixeldisparity map may be reconfigured according to the pattern evolution.Cellular automata are mathematical models for systems in which manysimple components interact with each other to produce complex patternsof behavior. Many schemes are available for 1D cellular automata.

According to an embodiment of the disclosure, 1D and 2D cellularautomata may use higher-order automata without restricting generality.The extension of 1D cellular automata to 2D cellular automata isimportant for extending the principle of the disclosure intended togenerate a higher-order pattern.

A cellular automaton includes a regular lattice of sites (or cells).Each site takes on k possible values (species), and is updated indiscrete time steps according to a rule that depends on the value ofsites in some neighborhoods around it. The value a_(i) ^(t+1)(Equation 1) of a site at position i in 1D cellular automaton A1 dependson nearest neighbors and thus is determined by a rule that evolvesaccording to the following equation.

There are several possible lattices and neighborhood structures for 2Dcellular automata. A 5-neighbor cellular automaton evolves similarly by1D cellular automata according to [Equation 2].

Further, in a special class of totalistic rules, the value of a site maydepend on the sum of the values in the neighborhood.

a _(ij) ^(t+1)=(a _(i−1,j) ^(t) +a _(i,j) ^(t) +a _(i+1,j) ^(t) +a_(i,j−1) ^(t) +a _(i,j+1) ^(t))   [Equation 3]

FIGS. 5A and 5B illustrate neighborhood structures considered for 2Dcellular automata according to various embodiments of the disclosure.

Referring to FIGS. 5A and 5B, in cellular automaton evolution, thevalues of center cells 41 and 42 are updated according to a rule thatdepends on the values of shaded cells. A cellular automaton illustratedin FIG. 5A is referred to as a “five-neighbor square”, and a cellularautomaton illustrated in FIG. 5B is referred to as a “nine-neighborsquare”. These neighborhoods are sometimes referred to as the vonNeumann neighborhood and the Moore neighborhood, respectively.Totalistic cellular automata rules take the value of the center sitethat depends on the sum of the values of the sites in the neighborhood.With outer totalistic rules, sites are updated according to theirprevious values and the sum of the values of the other sites in theneighborhood. For example, triangular and hexagonal lattices are alsopossible, but they are not used in the examples given here.Five-neighbor square, triangular, and hexagonal cellular automata rulesmay all be considered special cases of the general neighbor squarerules.

FIG. 6 is a flowchart illustrating a method of generating a cellularautomaton pattern according to an embodiment of the disclosure.

First, the processor 430 determines the cell size for a cellularautomaton pattern in operation 5610. For example, the cell size for thecellular automaton pattern may be determined in pixels. For example, thecell size of the cellular automaton pattern may be determined to be 1×1,2×2, and 3×3 (pixel×pixel).

Then, the processor 430 determines a rule for generating the cellularautomaton pattern in operation S620, which will be described below indetail with reference to FIG. 8.

The processor 430 selects a row or column for which an initial value isto be set in operation S630. For example, the processor 430 may set theleftmost column of a 2D image with 3×3 pixels as the column for which aninitial value is to be set. The row or column for which an initial valueis to be set is determined to be at least one of the rows or columns ofthe 2D image.

The processor 430 may set the initial value in operation S640. Forexample, if the selected column for which an initial value is to be setincludes 10 cells, the initial value may be set to 0110011100. Herein,each digit of the determined initial value corresponds to one of thecells.

The processor 430 determines the value of a row or column next to theinitial row or column on the basis of the determined initial value andthe determined pattern generation rule in operation S650. The next rowor column to the initial row or column may be the row or column closestto the initial row or column.

The above operations S610 to 650 may be repeated until values are setfor all cells of the 2D image.

While operations S610 to S650 may be performed sequentially, all ofoperations S610 to S650 are not necessarily performed. For example, ifthe initial value is set, the other operations except for the initialvalue setting operation S640 may be performed. An embodiment ofgenerating a cellular automaton pattern will be described with referenceto FIGS. 7 to 17.

FIG. 7 illustrates a 2D image according to an embodiment of thedisclosure.

The processor 430 may determine the cell size for a cellular automatonpattern.

Referring to FIG. 7, the width and height of a 2D image 701 are set toIMAGE_WIDTH and IMAGE_HEIGHT, respectively. In addition, the 2D image701 includes a plurality of cells. For example, the 2D image 701 mayinclude cells of the same size. In this case, the width and height ofone cell 701-1 of the 2D image 701 may be set to CELL_SIZE. As describedbefore, the size of one cell 701-1 in the 2D image 701 may be set inpixels.

The processor 430 may configure a rule for generating a cellularautomaton pattern. The cellular automaton generation rule may be ascheme for determining the value of another cell neighboring a centercell of three contiguous cells on the basis of the values (or states) ofthe three contiguous cells. The three contiguous cells are included inone row or column.

FIG. 8 illustrates a rule for generating a cellular automaton patternaccording to an embodiment of the disclosure.

To facilitate the description of the embodiment of the disclosure, apattern generation rule is referred to as Rule #25. However, the milefor generating a cellular automaton pattern may also be referred to byany other name. Further, various rules may be available for generating acellular automaton pattern, and are not limited to the following rule.

Referring to a table 801 illustrated in FIG. 8, s0 and s1 are values ofa cell at the center of three contiguous cells selected from the 2Dimage 701. k0, k1, and k2 are sums of values of two cells on both sidesof the center cell. Values 801-1, resulting from s0, s1, k0, k1, and k2,are values of a cell neighboring the center cell.

S may be defined by generalizing S0 and S1 according to [Equation 4].

s=S_(i)   [Equation 4]

Further, the sum of the values of two cells neighboring a cell that hasthe defined value s is defined as k by [Equation 5]. The cell that hasthe defined value s and the two neighoring cells may be contiguous inone row or column. In another example, the sum of two cells apart fromthe cell that has the defined value s by a predetermined number of cellsmay be defined as k.

$\begin{matrix}{k = {\sum\limits_{{k = j},{i \neq j}}^{neighborhood}S_{i}}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

The result of applying the determined values s and k to Rule #25 801-1may be calculated by [Equation 6].

$\begin{matrix}{S_{i}^{{next}\mspace{14mu} {epoch}} = {{bin}_{s,k} = {bin}_{S_{i},{\sum\limits_{j,{i \neq j}}^{neighborhood}S_{j}}}}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

For example, referring to FIG. 9, the processor 430 may select threecontiguous cells, Cell(i−1), Cell(i), and Cell(i+1) in a column of the2D image 701. Referring to FIG. 10, the three contiguous cells,Cell(i−1), Cell(i), and Ce11(i+1) are denoted by reference numerals10-2, 10-1, and 10-3, respectively, and may have values of 1, 0, and 0,respectively. In this case, s is 0, and K is the sum of 1 and 0, thatis, 1. Then, the result of applying s and k to Rule #25 801-1, S_(i)^(next epoch) is 0 according to [Equation 7].

$\begin{matrix}{{S_{i}^{{next}\mspace{14mu} {epoch}} = {{0\mspace{14mu} {if}\mspace{14mu} s} = \left. 0\rightarrow{s\; 0} \right.}},{k = {{\sum\limits_{j,{i \neq j}}^{2}S_{j}} = \left. 1\rightarrow{k\; 1} \right.}}} & \left\lbrack {{Equation}\mspace{14mu} 7} \right\rbrack\end{matrix}$

The result 0 is set as the value of a cell 10-4.

According to an embodiment, Rule #25 801-1 may be defined by [Equation8].

With reference to FIGS. 11 to 18, a detailed description will be givenof a method of generating a pattern on the basis of cellular automatongeneration.

FIG. 11 is a view depicting a method of setting an initial valueaccording to an embodiment of the disclosure.

According to an embodiment of the disclosure, it is assumed that a 2Dplane includes 9×9 cells, that is, 9 cells in a horizontal direction by9 cells in a vertical direction. 9×9 may be determined according to thesize of a detector that detects a pattern reflected from an object.

Referring to FIG. 11, the processor 430 selects one of the columns ofcells included in a 2D plane, as indicated by reference numeral 111. Theprocessor 430 may set 110101111 as an initial value for each of thecells of the selected column, as indicated by reference numeral 112.FIG. 12 illustrates the result of setting an initial value according toan embodiment of the disclosure.

FIGS. 13 to 17 illustrate a detailed cellular automaton processaccording to an embodiment of the disclosure.

FIG. 13 illustrates a case in which index i of the uppermost cell 132 (sin Rule #25) of a selected column in a 2D image is set to 0 according toan embodiment of the disclosure. In other words, the processor 430determines the cell 132 and neighbor cells 131 and 133 of the cell 132to be initial targets to which Rule #25 is to be applied.

The value (or state) of the cell 132 in the middle of the threecontiguous cells 131, 132, and 133 is 1. Further, the sum k of thevalues of the neighbor cells 131 and 133 of the cell 132 is 2.Application of s and k to Rule #25 results in an updated value (orstate) 135 that is 0.

FIG. 14 illustrates an operation for index i that is 1, subsequent tothe operation in FIG. 13. According to an embodiment of the disclosure,the value of a cell 142 in the middle of three contiguous cells 141,142, and 143 is 1. In addition, the sum k of the values of the cells 141and 143 neighboring the cell 142 is 1. An updated value 145 resultingfrom application of s and k to Rule #25 is 1.

FIG. 15 illustrates an operation for index i that is 2, subsequent tothe operation in FIG. 14. According to an embodiment of the disclosure,the value of a cell 152 in the middle of three contiguous cells 151,152, and 153 is 0. In addition, the sum k of the values of the cells 151and 153 neighboring the cell 152 is 2. An updated value 154 resultingfrom application of s and k to Rule #25 is 1.

(a) of FIG. 16 illustrates a column 162 updated on the basis of a column161 for which an initial value has been set. The contiguous cells of theupdated column 162 have a value of 011011000. Further, the processor 430may update the value of a cell 163-1 in the column 163 to 0 on the basisof updated values of contiguous cells 162-1, 162-2, and 162-3 in theupdated column 162.

(b) of FIG. 16 illustrates the result of updating the value of a cell163-2 on the basis of three contiguous cells 162-4, 162-5, and 162-6.(c) of FIG. 16 illustrates the result of updating the value of a cell163-3 on the basis of three contiguous cells 162-7, 162-8, and 162-9.(d) of FIG. 16 illustrates the result of updating the value of a cell163-3 on the basis of three contiguous cells 162-10, 162-11, and 162-12.

FIG. 17 illustrates a result of implementing the foregoing embodiment onthe basis of the operations illustrated in FIGS. 13 to (d) of 16.Referring to FIG. 17, it may be noted that the values of all cellsincluded in a 2D image have been updated.

The 2D image 101 is a pattern obtained by performing cellular automatonaccording to an embodiment of the disclosure. Since a self-pattern maybe formed with a cellular automaton according to a predetermined rule,wasting memory resources and communication resources may be prevented.

FIG. 18 is a view depicting a method of determining the pitch size foran electromagnetic field filter, and the cell size for generation of acellular automaton according to an embodiment of the disclosure.

It is assumed that parameters IMAGE_HEIGHT and IMAGE_WIDTH, whichrepresent the size of a 2D image, are preset.

Each angular resolution ARESc 182 f a lens system in a detector using aviewing angle parameter FOVc is determined by [Equation 9].

ARESc=CEE_(SIZE)*FOVc/√{square root over (DETECTOR_HEIGHT²+DETECTOR₁₃WIDTH²)}  [Equation 9]

Each angular resolution, ARESp of a filter using a viewing angleparameter Vp is determined by [Equation 10].

ARESp=PITCH*FOVp/√{square root over(FILTER_HEIGHT²+FILTER_WIDTH²)}  [Equation 10]

If ARESc and ARESp are equal, the pitch size 181 for an electromagneticfield filter is determined on the basis of [Equation 11] according to[Equation 9] and [Equation 10].

$\begin{matrix}{\frac{PITCH}{{CELL}_{SIZE}} = {\frac{\sqrt{{FILTER\_ HEIGHT}^{\; 2} + {FILTER\_ WIDTH}^{\; 2}}}{\sqrt{{DETECTOR\_ HEIGHT}^{\; 2} + {DETECTOR\_ WIDTH}^{\; 2}}}*{FOVc}\text{/}{FOVp}}} & \left\lbrack {{Equation}\mspace{14mu} 11} \right\rbrack\end{matrix}$

Further, the cell size for the cellular automaton is determined by[Equation 12].

IMAGE_HEIGHT=CELL_SIZE/PITCH*FILTER_HEIGHT;

IMAGE_WIDTH=CELL_SIZE/PITCH*FILTER_WIDTH;   [Equation 12]

In addition, the processor 430 may map the positions of cellularautomaton cells according to the filter pitch size so that the center ofa feature (I, J) of a filter array is equal to(j*IMAGE_WIDTH/CELL_(SIZE)+I)*FILTER_HEIGHT/PITCH.

Further, the processor 430 may design the shape of a pattern accordingto a modification to the size of a diffracted image at a corner of arectangle, as indicated by reference numeral 183. In this case, theradius of the corner may be set to 1.22*PITCH.

FIG. 19 illustrates a 2D image according to an embodiment of thedisclosure.

Referring to FIG. 19, the 2D image 191 is an image for which a pixelconfiguration has been considered. For example, a first cell in the topleft position in FIG. 17, having a value of 1, may include 4 pixels (2×2pixels). To represent the value of 1 for the first cell in the top leftposition in FIG. 17, all of pixels 191-1 to 191-4 may be 1 in the 2Dimage 191.

FIG. 20 illustrates a 2D image according to another embodiment of thedisclosure.

Referring to FIG. 20, a 2D image 201 includes 18×18 pixels. In the 2Dimage 201, one cell may be represented by four pixels. For example, fourpixels 201-1 included in the 2D image 201 may represent the value 1 of afirst cell in the top left position in FIG. 17. In another example, fourother pixels 201-2 included in the 2D image 201 may represent the value0 of a second cell to the left of the first cell located in the top leftposition in FIG. 17.

FIG. 21 illustrates a process of projecting and capturing a cellularautomaton pattern according to an embodiment of the disclosure.

FIG. 21 illustrates an embodiment in which a pattern 211 generated witha cellular automaton (hereinafter, referred to as a cellular automatonpattern) is projected onto a plane with a zero depth disparity across atotal area. The depth disparity may be defined as the distance between alight projector that projects a cellular automaton pattern and theplane.

Referring to FIG. 21, the cellular automaton pattern 211 includes apattern generated with a cellular automaton. As light projected by thelight projector passes through a filter or mask, the cellular automatonpattern 211 may be generated,

For example, the processor 430 may project the cellular automatonpattern 211 onto the plane with zero depth disparity across the totalarea by means of the light projector and the filter or mask.

In this case, a reflected cellular automaton pattern 212 has the depthparity of 0 across its total area, and thus may be identical to thecellular automaton pattern 211.

FIG. 22 illustrates a process of projecting and capturing a cellularautomaton pattern according to another embodiment of the disclosure.

In the illustrated case of FIG. 22, a cellular automaton pattern isprojected to a 3D space with a non-zero depth disparity.

A light projector for projecting a cellular automaton pattern 221 may beset apart from a camera (a detector or a light receiving unit) forcapturing a pattern reflected from a space by a predetermined distance.For example, the light projector and the camera may be provided on onesurface of a terminal, set apart from each other by a predetermined gap.Thus, the angle at which the cellular automaton pattern 221 is projectedfrom the light projector onto an object in a 3D space is different fromthe angle at which the cellular automaton pattern 221 is reflected fromthe object and received at the camera. Therefore, the cellular automatonpattern 221 reflected from the object may be shifted in a predetermineddirection.

For example, the processor 430 may project the cellular automatonpattern 221 to the 3D space. Herein, the captured pattern 222 of thecellular automaton pattern 221 reflected from the object may be shiftedin a predetermined direction (parallax shift in FIG. 22). Herein, thesize of the area 222-1 from which the cellular automaton pattern 221 hasbeen shifted is depth information between the camera and the object.

FIG. 23 is a flowchart illustrating a method of decoding a cellularautomaton pattern according to an embodiment of the disclosure.

The processor 430 captures the cellular automaton pattern that has beenprojected and then reflected in operation S2310. Further, the processor430 determines the values of cells included in the cellular automatonpattern by clustering the captured cellular automaton pattern, and thenrecognizes the cellular automaton pattern in operations S2320 and S2330.For example, the value of each cell in the cellular automaton patternmay be determined to be 0 or 1. In another example, the value of eachcell in the cellular automaton pattern may be selected from among 0 andpositive integers. The processor 430 detects horizontal and verticalconnected components of the clustered pattern in operation S2340. Theprocessor 430 estimates depth disparity on the basis of the horizontaland vertical connected components in operation S2350. The processor 430performs object segmentation on the basis of the estimated depthdisparity. The processor 430 generates a 3D map in operation S2370.

FIGS. 24A and 24B illustrate a clustering method according to anembodiment of the disclosure.

FIG. 24A, the left pattern is an example of a captured cellularautomaton pattern. The captured cellular automaton pattern has pixelvalues (0 to 255 or above).

The processor 430 may perform clustering on the basis of the clusterwindow CLUSTER WINDOW of a predetermined size. Referring to the leftpattern of FIG. 24A, the processor 430 performs clustering on the basisof a 4×4 cluster window 2*CELL_SIZE*2*CELL_SIZE.

Referring to FIG. 24B, the processor 430 may cluster the value of eachpixel within the cluster window to 0 or 1 by a clustering algorithm(e.g., a K-means algorithm).

The right pattern in FIG. 24A is the clustered result. Herein, a part242 of the values within the cluster window may be different from valuesof the projected cellular automaton pattern. This error may be caused bypixel blending, data loss, and so on.

FIGS. 25A to 25E illustrate clustered results according to an embodimentof the disclosure.

FIG. 25A illustrates the result of capturing a cellular automatonpattern with each cell mapped to 2×2 pixels and clustering the capturedcellular automaton pattern. For example, four pixel values 252 to 254may be mapped to one cell.

FIG. 25B illustrates the values of the top left pixels of the cells,each mapped to four pixels, FIG. 25C illustrates the values of the topright pixels of the cells, each mapped to four pixels, FIG. 25Dillustrates the values of the bottom left pixels of the cells, eachmapped to four pixels, and FIG. 25E illustrates the values of the bottomright pixels of the cells, each mapped to four pixels.

FIGS. 26A to 26F illustrate a method of recognizing a cellular automatonpattern according to an embodiment of the disclosure.

According to an embodiment, the method of recognizing a cellularautomaton pattern uses the foregoing method of generating a cellularautomaton pattern.

FIG. 26A illustrates a clustered cellular automaton pattern 261. Theclustered cellular automaton pattern 261 may include a shadow area 263due to object-incurred shifting.

The processor 430 may recognize the clustered cellular automaton pattern261 by comparing the clustered cellular automaton pattern 261 with apattern generated with a cellular automaton on the basis of an initialvalue.

For example, the processor 430 may perform a cellular automaton processby using the clustered value 110101111 of a first column 261-1 in thecellular automaton pattern 261 as an initial value. Herein, the cellularautomaton process is identical to the foregoing example of cellularautomaton pattern generation. The processor 430 may obtain a value of 0by applying values 261-2, 261-3, and 261-4 of the first column 261-1,which are 1 in the cellular automaton pattern 261 to Rule #25. Theprocessor 430 may compare the obtained value 261-5, which is 0 (it maybe stored in a temporary array of the memory), with a clustered value262-6 which is 0.

Then, the processor 430 in FIG. 2613 may obtain a value 261-1, which is1, by applying values 1, 1 and 0 (in a highlighted area) of the firstcolumn 261-1 in the cellular automaton pattern 261 to Rule #25. Theprocessor 430 may compare the obtained value 261-10, which is 1, with aclustered value 262-11 which is 0.

FIG. 26C illustrates the result 261-12 of repeating the above-describedoperation for the first column 261-1 of the cellular automaton pattern261.

FIG. 26D illustrates a method of comparing the result 261-12 with asecond column 261-13 of the cellular automaton pattern 261. For example,since the first value 0 of the result 261-12 is equal to the first value0 of the second column 261-13, the processor 430 sets the value of thefirst cell of a current column 261-14 to 0.

Then, as illustrated in FIG. 26E, since the second value 1 of the result261-12 is equal to the second value 0 of the second column 261-13, theprocessor 430 sets the value of the second cell of the current column261-14 to 0.

FIG. 26F illustrates the result of applying the operations described inreference to FIGS. 26A to 26E to up to the 8^(th) column 261-15 of thecellular automaton pattern 261.

FIG. 27 illustrates indexes of a cellular automaton pattern according toan embodiment of the disclosure. Specifically, FIG. 27 illustrates avertical disjoint set 271 and a horizontal disjoint set 272 for cellindexes of a cellular automaton pattern.

FIG. 28 illustrates a method of processing indexes of a cellularautomaton pattern according to an embodiment of the disclosure.Specifically, FIG. 28 illustrates a process of sequentially insertingthe indexes of the first column of a cellular automaton pattern 273 intoa memory stack 274.

FIGS. 29A to 29H illustrate a method of detecting horizontal andvertical connected components according to an embodiment of thedisclosure.

Referring to FIG. 29A, the processor 430 acquires order 8 of a cell271-1 from a stack 274. Further, the processor 430 may acquire index iof the cell 271-1, that is, 72 (=8×9) by [Equation 13].

 i = n  DETECTOR WIDTH CELL SIZE + C i ,  ( C i   column   index, o ≤ C i < DETECTOR WIDTH CELL SIZE ) [ Equation   13 ]

The processor 430 detects cells 271-1 and 271-2 with index 72 from thevertical disjoint set 271 and the horizontal disjoint set 272,respectively. Further, the processor 430 defines, as 72, a parent indexfor the vertical disjoint set 271 and the horizontal disjoint set 272.The processor 430 merges the cell with index 72 in the vertical disjointset 271 with up to a cell with parent index 72. The processor 430 mergesthe cell with index 72 in the horizontal disjoint set 272 with up to acell with index 73 (=72+1).

The processor 430 may repeat the operation described with reference toFIG. 29A for the next cell. Referring to FIG. 29B, the processor 430acquires order 7 of the cell 271-1 from the stack 274. In addition, theprocessor 430 may acquire index i of the cell 271-1, that is, 63 (=7×9)by [Equation 13]. The processor 430 detects cells 271-3 and 271-4 withindex 63 from the vertical disjoint set 271 and the horizontal disjointset 272, respectively. Further, the processor 430 defines, as 72, theparent index for the vertical disjoint set 271 and the horizontaldisjoint set 272. The processor 430 merges the cell with index 63 in thevertical disjoint set 271 with up to the cell with parent index 72. Theprocessor 430 merges the cell with index 63 in the horizontal disjointset 272 with up to a cell with index 64 (=63+1). The result ofcompleting the above operation for the first columns of the verticaldisjoint set 271 and the horizontal disjoint set 272 is illustrated inFIG. 29C.

FIG. 29D illustrates a method of detecting horizontal and verticalconnected components in the first to third cells of the second column(Ci=1), and FIG. 29E illustrates the result of detecting the horizontaland vertical connected components in the first to third cells of thesecond column (Ci=1).

FIG. 29F illustrates the result of performing the method of detectinghorizontal and vertical connected components, for the 8^(th) column(Ci=7) of the cellular automaton pattern 273.

FIG. 29G illustrates the borders of the vertical disjoint set 271 andthe horizontal disjoint set 272 according to an embodiment of thedisclosure.

FIG. 29H illustrates borders of the cellular automaton pattern 273according to an embodiment of the disclosure.

FIGS. 30A to 30H illustrate a method of estimating the disparity for avertical cluster according to an embodiment of the disclosure.

The processor 430 may estimate the disparity between a projectedcellular automaton pattern (hereinafter, referred to as an originalpattern) and a reflected and captured cellular automaton pattern(hereinafter, referred to as a captured pattern) of the projectedcellular automaton pattern. Disparity estimation as described below maybe performed for all of the remaining recognized cells except forbordering cells of a cellular automaton pattern, described withreference to FIG. 29H.

FIG. 30A illustrates an original pattern 301, and a non-shifted capturedpattern 302. Referring to FIG. 30A, the processor 430 may calculate thedifferences between all cells of the original pattern 301 and thecaptured pattern 302. For example, the processor 430 may record (orstore) the difference 303-1 as being 0 between the value 1 of a firstcell 301-1 of the original pattern 301 and the value 1 of a first cell302-1 of the captured pattern 302.

Referring to FIG. 30B, the processor 430 may determine the differencevalues of a cluster 304 of connected cells on the basis of the result303 of calculating the differences. For example, if a first column 303-1of the difference result 303 is connected, the processor 430 may set thesum of the values of cells in the first column 303-1 as the differencevalues of cells in a first column 304-1 of the cluster 304. Therefore,the differences of cells in the first column 304-1 of the cluster 304are set to 0.

Referring to FIG. 30C, for a second column 303-2 of the differenceresult 303, the processor 430 may determine the sum of the values of theremaining cells except for bordering cells to be cluster differencesbetween the original pattern 301 and the captured pattern 302.

Referring to FIG. 30D, for a third column 303-3 of the difference result303, the processor 430 may determine the sum of the values of theremaining cells, except for bordering cells, to be cluster differencesbetween the original pattern 301 and the captured pattern 302.

Referring to FIG. 30E, for a sixth column 303-4 of the difference result303, the processor 430 may determine the sum of the values of theremaining cells, except for the bordering cells, to be clusterdifferences between the original pattern 301 and the captured pattern302.

Referring to FIG. 30F, for a seventh column 303-5 of the differenceresult 303, the processor 430 may determine the sum of the values of theremaining cells, except for the bordering cells, to be clusterdifferences between the original pattern 301 and the captured pattern302.

Referring to FIG. 30G, for an eighth column 303-6 of the differenceresult 303, the processor 430 may determine the sum of the values of theremaining cells, except for the bordering cells, to be clusterdifferences between the original pattern 301 and the captured pattern302.

FIG. 30H illustrates cluster difference values between the originalpattern 301 and the captured pattern 302. Herein, connected cells havethe same difference value.

FIGS. 31A to 31D illustrate a method of estimating the disparity for ahorizontal cluster according to an embodiment of the disclosure.

FIG. 31A illustrates the original pattern 301 and the non-shiftedcaptured pattern 302.

Referring to FIG. 31A, the processor 430 may determine difference valuesof a cluster 306 of connected cells on the basis of a difference result305. For example, if a first row 305-1 of the difference result 305 isconnected, the processor 430 may set the sum of the values of cells inthe first row 305-1 as the difference values of cells in a first row305-1 of a cluster 306. Therefore, the differences of cells in the firstrow 305-1 of the cluster 304 are set to 0.

Referring to FIG. 31B, for the first, third, fifth, seventh, and ninthrows of the difference result 305, the processor 430 may determine thesum of the values of the cells inside clusters except for the borderingcells to determine the difference sum between the original pattern 301and the captured pattern 302. Therefore, the sum of values of cellsinside of clusters 304 of said rows in the difference result 305 isdetermined to 0. Referring to FIG. 31C, an another example of clustersis depicted for the first, fourth, sixth, and eighth rows of thedifference result 305, the processor 430 may determine herein thedifference sum between the original pattern 301 and the captured pattern302 as 4, due to the fourth and sixth rows.

FIG. 31D illustrates cluster difference values 307 between the originalpattern 301 and the captured pattern 302. Herein, connected cells havethe same difference value except the fourth and the sixth rows.

FIGS. 32A and 32B illustrate a method of determining the sum of avertical cluster difference and a horizontal cluster differenceaccording to an embodiment of the disclosure.

Referring to FIG. 32A, the processor 430 may determine the sums ofdifference values by comparing all rows of the result 307 of horizontalcluster differences with all rows of the result 304 of vertical clusterdifferences. The processor 430 may not determine the sums of differencevalues for bordering cells among contiguous cells. For example, theprocessor 430 may determine the difference value of a cell 321-1 inwhich the first row of the result 307 of horizontal cluster differencesmeets with the second column of the result 304 of vertical clusterdifferences. The value of the overlapped cell 321-1 may be the sum ofthe difference value 0 of the first row of the result 307 of horizontalcluster differences and the difference value 0 of the second column ofthe result 304 of vertical cluster differences.

Referring to FIG. 32B, the processor 430 may determine the sum 4 of thedifference value 2 of the sixth row of the result 307 of horizontalcluster differences and the difference value 2 of the sixth column ofthe result 304 of vertical cluster differences to be the differencevalue of a cell 321-1 in which the sixth row of the result 307 ofhorizontal cluster differences meets with the sixth column of the result304 of vertical cluster differences.

FIG. 32B illustrates a table 308 resulting from the operation ofdetermining the sum of difference values for every row of the result 307of horizontal cluster differences and every column of the result 304 ofvertical cluster differences.

FIGS. 33A and 33B illustrate a method of determining a shift tableaccording to an embodiment of the disclosure.

The processor 430 may generate a shift table by using the result table308 illustrated in FIG. 32B.

For example, referring to FIG. 33A, the processor 430 may determine aminimum difference value by comparing the value of every cell of theresult table 308 with the value of a cell in an uncertainty initialtable 309. In this case, [Equation 14] may be used.

Argmin_(shift)(difference_(horizontal)+difference_(vertical))*

Herein, cells that are compared with each other may be at the sameposition in the respective tables. Further, if the value of a specificcell is not set, the processor 430 may determine the comparison resultfor the specific cell to be 0.

For example, the processor 430 may compare the value of a cell 308-1 inthe result table 308 with the value of a cell 309-1 in the initial table309. For example, since the value of the cell 309-1 in the initial table309 is not set, min(4, −) for the value 4 of the cell 308-1 in theresult table 308 and the value of the cell 309-1 in the initial table309 may be determined to be 4. Meanwhile, the processor 430 may notperform the cell value comparison operation for the bordering cellsillustrated in FIG. 29H.

A minimum cell difference table 310 illustrated in FIG. 33B listsminimum values between the values of cells in the result table 310 andthe values of cells in an initial table 311 on the basis of the cellvalue comparison method of FIG. 33A. If the minimum difference of a cellis changed in the minimum cell difference table 310, the processor 430may update the shift amount. For example, if the processor 430determines that the value 4 of a cell 310-1 of the minimum celldifference table 310 has been changed, the processor 430 may update thecurrent shift amount 311-1, which is 0.

FIGS. 34A to 34G illustrate a method of estimating the disparity for avertical cluster according to another embodiment of the disclosure.

A description redundant with the description of FIGS. 30A to 30H willnot he provided herein.

FIGS. 34A to 34G illustrate an original pattern 341, and a pattern 342obtained by shifting one captured pattern by one cell.

Referring to FIG. 34A, the processor 430 may determine the differencevalues between the values of cells in the first column of the originalpattern 341 and the values of cells in the second column of the shiftedpattern 342. In this case, the processor 430 may determine thedifference values between cells in the same rows among the cells of thefirst column of the original pattern 341 and the cells of the secondcolumn of the shifted pattern 342. Further, the processor 430 may recordthe determined difference values in the rows of a first column 343-1 ofa table 343.

Referring to FIG. 34B, the processor 430 may determine the differencevalues between the values of cells in the second column of the originalpattern 341 and the values of cells in the third column of the shiftedpattern 342. The processor 430 may record the determined differencevalues in the second column 343-2 of the table 343.

Referring to FIG. 34C, the processor 430 may determine the differencevalues between the values of cells in the second and eighth column ofthe original pattern 341 and the values of cells in the ninth column ofthe shifted pattern 342. The processor 430 may record the determineddifference values in the rows of an eighth column 343-2 of the table343.

A table 345 illustrated in FIG. 34D may be obtained by shifting theposition of the ninth column of the table 343 illustrated in FIG. 34C tothe position of the first column.

A table 346 illustrated in FIG. 34E includes results of accumulatingdifference values for vertical clusters on the basis of the table 345 ofFIG. 34D. Herein, the method of accumulating difference values forvertical clusters has been described in detail with reference to FIGS.30A to 30H, and thus will not be described herein.

A table 347 illustrated in FIG. 34F includes results of accumulatingdifference values for horizontal clusters on the basis of the table 345of FIG. 34D. Herein, the method of accumulating difference values forhorizontal clusters has been described in detail with reference to FIGS.31A to 31D, and thus will not be described herein.

The processor 430 may sum difference values by comparing all of the rowsof vertical cluster difference values 346 with all of the rows ofhorizontal cluster difference values 347. The summation of differencevalues has been described in detail with reference to FIGS. 32A and 32B,and thus will not be described herein.

For example, FIG. 34G illustrates a table 348 including the resultvalues of the summation of difference values obtained on the basis ofthe vertical cluster difference values 346 and the horizontal clusterdifference values 347.

FIGS. 35A and 35B illustrate a method of determining a shift tableaccording to another embodiment of the disclosure.

The processor 430 may generate a shift table by using the result table348 illustrated in FIG. 34G.

For example, referring to FIG. 35A, the processor 430 may determine theminimum difference values by comparing the values of all cells in theresult table 348 with the values of all cells in the minimum celldifference table 310 illustrated in FIG. 33A. In this case, [Equation14] may be used. Cells that are compared with each other may be at thesame position in the respective tables. Further, if the value of aspecific cell is not set, the processor 430 may determine the comparisonresult for the specific cell to be 0.

For example, the processor 430 may determine min(2,4) for the value 2 ofa cell 351-1 in the result table 348 and the value 4 of a cell 310-11 inthe minimum cell difference table 310 to be 2.

Further, the processor 430 may not perform the cell value comparisonoperation, for example, for the bordering cells illustrated in FIG. 29H.

A minimum cell difference table 352 in FIG. 35B is the result ofcomparing the values of the cells in the result table 348 with thevalues of the cells in the minimum cell difference table 310 in FIG.33A, on the basis of the cell value comparison method described withreference to FIG. 35A.

If the minimum difference of a cell is changed in the minimum celldifference table 352, the processor 430 may update the shift amount. Forexample, if the processor 430 determines that the value 2 of a cell352-1 of the minimum cell difference table 352 has been changed, theprocessor 430 may update the current shift amount 353-1, which is 1.

FIGS. 36A, 36B and 36C illustrate a method of determining a shift tableaccording to another embodiment of the disclosure.

FIG. 36A illustrates a table 361 including the result values of thesummation of difference values calculated on the basis of verticalcluster difference results and horizontal cluster difference resultswhen a captured pattern is shifted by two cells.

Referring to FIG. 36B, the processor 430 may determine the minimumdifference values by comparing the values of all of the cells in thetable 361 with the values of all of the cells in the minimum celldifference table 352 illustrated in FIG. 35B. In this case, [Equation14] may be used. Cells that are compared with each other may be at thesame position in the respective tables. Further, if the value of aspecific cell is not set, the processor 430 may determine a comparisonresult for the specific cell to be 0.

For example, the processor 430 may determine min(3, 2) for the value 3of a cell 362-1 in the table 362 and the value 2 of a cell 352-11 in theminimum cell difference table 352 to be 2.

Further, the processor 430 may not perform the cell value comparisonoperation, for example, for the bordering cells illustrated in FIG. 29H.

A minimum cell difference table 363 in FIG. 36C is the result ofcomparing the values of the cells in the table 362 with the values ofthe cells in the minimum cell difference table 352, on the basis of thecell value comparison method described with reference to FIG. 36B.

If the minimum difference of a cell is changed in the minimum celldifference table 363, the processor 430 may update the shift amount. Forexample, if the processor 430 determines that the value 2 of a cell363-1 of the minimum cell difference table 363 has not been changed, theprocessor 430 may not additionally update the current shift amount forthe cell 364-1.

A result table 363 in FIG. 36C may be the result of comparing the valuesof the cells in the result table 361 with the values of the cells in theminimum cell difference table 310 of FIG. 35A on the basis of the cellvalue comparison method described with reference to FIG. 35A.

If the minimum difference of a cell is changed in the minimum celldifference table 363, the processor 430 may update the shift amount. Forexample, if the processor 430 determines that the value 2 of the cell363-1 of the minimum cell difference table 363 has been changed, theprocessor 430 may update the current shift amount 364-1, which is 2.

FIGS. 37A, 37B and 37C illustrate a method of determining a shift tableaccording to another embodiment of the disclosure.

FIG. 37A illustrates a table 371 including the result values ofsummation of difference values calculated on the basis of verticalcluster difference results and horizontal cluster difference result whena captured pattern is shifted by three cells.

Referring to FIG. 37B, the processor 430 may determine minimumdifference values by comparing the values of all of the cells in thetable 361 with the values of all of the cells in the minimum celldifference table 363 illustrated in FIG. 36C. In this case, [Equation14] may be used. Cells which are compared with each other may be at thesame position in the respective tables. Further, if the value of aspecific cell is not set, the processor 430 may determine a comparisonresult for the specific cell to be 0.

For example, the processor 430 may determine min(0, 2) for the value 0of a cell 372-1 in the table 372 and the value 2 of a cell 363-1 in theminimum cell difference table 363 to be 0.

Further, the processor 430 may not perform the cell value comparisonoperation, for example, for the bordering cells illustrated in FIG. 29H.

A minimum cell difference table 373 in FIG. 37C is the result ofcomparing the values of the cells in the table 372 with the values ofthe cells in the minimum cell difference table 363, on the basis of thecell value comparison method described with reference to FIG. 37B.

If the minimum difference of a cell is changed in the minimum celldifference table 373, the processor 430 may update the shift amount. Forexample, if the processor 430 determines that the value 0 of a cell373-1 of the minimum cell difference table 373 has been changed, theprocessor 430 may update by the current shift amount 3 for the cell374-1.

Meanwhile, the processor 430 may process bordering cells.

To process a bordering cell, the processor 430 may use a per-componentdifference between an original pattern and a shifted and clusteredpattern. Further, the processor 430 may use an accumulated difference ofeach cell for every vertical cluster. Further, the processor 430 may usean accumulated difference of each cell for every horizontal cluster.Further, the processor 430 may use an updated shift for all cells exceptbordering cells. Further, the processor 430 may process type-1horizontal bordering cells, type-2 horizontal bordering cells, type-1vertical bordering cells, and type-2 vertical bordering cells,separately or together.

An example of processing a bordering cell will be described below withreference to FIGS. 38A to 41E.

FIGS. 38A to 38E illustrate a method of processing a bordering cellaccording to an embodiment of the disclosure.

FIGS. 38A to 38E illustrate processing type-1 horizontal borderingcells. Type-1 cells may be defined as cells in a vertical cluster whichhas horizontal and vertical connections, a cell index identical to theindex of a horizontal cluster, and/or at least one cell of type 1.

FIGS. 38A to 38D illustrate type-1 and/or type-2 bordering cells in thecases of 0-cell shift, 1-cell shift, 2-cell shift, and 3-cell shift,respectively.

FIG. 38E may have its horizontal and vertical clusters by indexesaccording to an example. Further, a cell 381-1 of a table 381 isgenerated when two or more shifts provide a difference of 0. However,the largest shift may be applied.

FIGS. 39A to 39E illustrate a method of processing a bordering cellaccording to another embodiment of the disclosure.

A description redundant with the description of FIGS. 38A to 38E willnot be provided herein.

FIGS. 39A to 39E illustrate processing type-2 horizontal borderingcells.

FIGS. 39A to 39D illustrate type-1 and/or type-2 bordering cells in thecase of 0-cell shift, 1-cell shift, 2-cell shift, and 3-cell shift,respectively.

In FIG. 39E, for example, if a cell is connected to its horizontalcluster and a left neighbor cell by indexes, the cell has a verticalcluster by the index of the left neighbor cell.

FIGS. 40A to 40E illustrate a method of processing a bordering cellaccording to another embodiment of the disclosure.

A description redundant with the description of FIGS. 38A to 39E willnot be provided herein.

FIGS. 40A to 40E illustrate processing type-1 vertical bordering cells.

FIGS. 40A to 40D illustrate type-1 and/or type-2 bordering cells in thecase of 0-cell shift, 1-cell shift, 2-cell shift, and 3-cell shift,respectively. Herein, a type-1 cell may be defined as a cell having ahorizontal connection.

In FIG. 40E, for example, once horizontal bordering cells are processed,the processed cells may be included.

If a cell is connected to its horizontal cluster and a left neighborcell by indexes, the cell has a vertical cluster by the index of theleft neighbor cell.

FIGS. 41A to 41E illustrate a method of processing a bordering cellaccording to another embodiment of the disclosure.

A description redundant with the description of FIGS. 38A to 40E willnot be provided herein.

FIGS. 41A to 41E illustrate processing type-2 vertical bordering cells.

FIGS. 41A to 41D illustrate type-1 and/or type-2 bordering cells in thecase of 0-cell shift, 1-cell shift, 2-cell shift, and 3-cell shift,respectively. Herein, a type-2 cell may he defined as a cell that doesnot have a horizontal connection but that has one or more neighborvertical clusters.

Referring to FIG. 40E, the processor 430 may detect an optimum shiftthat minimizes the differences among an original pattern, a clusteredpattern, and a neighbor cluster.

Referring to FIG. 41E, if two clusters provide an equivalent differenceof 0, a cell may be set as a bordering cell.

FIG. 42 illustrates object segmentation according to an embodiment ofthe disclosure.

Object segmentation is used to represent connected parts of the samedepth in an object. Disjoint sets and an estimated disparity map areused to detect object segmentation. Object segmentation may be performedwhen cellular automaton pattern recognition and a connected componentprocedure are performed immediately.

Referring to FIG. 42, the forehead part 421-1 of a human head-shapedobject 421 may correspond to the part 422-1 of a horizontal disjoint set422.

FIG. 43 is a flowchart illustrating an object segmentation methodaccording to an embodiment of the disclosure.

The processor 430 determines connected components of a vertical disjointset and a horizontal disjoint. Further, the processor 430 determines adisparity map in operation S4310.

The processor 430 may amend vertical clusters of the vertical disjointset using horizontal clusters from the horizontal disjoint set and theestimated disparity map in operation S4320.

The amendment is performed starting from a first cell algorithm and thenmoving from the left to right and up to down. In addition, if currentcells and/or cells have an equal shift, the current cell of the verticaldisjoint set is amended to have the index of its left neighbor cell.

The processor 430 stores the vertical disjoint set in the memory inoperation S4330.

The processor 430 amends the disjoint set according to a change in thedisparity map in operation S4340.

FIGS. 44A to 44H are views depicting an object segmentation methodaccording to an embodiment of the disclosure.

FIG. 44A illustrates a disparity map 441 according to an embodiment ofthe disclosure. FIG. 44B illustrates a vertical disjoint set 442 and ahorizontal disjoint set 443 according to an embodiment of thedisclosure.

The processor 430 determines (or reads) information about the disparitymap 441, the vertical disjoint set 442, and the horizontal disjoint set443.

For example, referring to FIGS. 44C and 44D, the processor 430 may amendindex information of at least the second column of the vertical disjointset 442 on the basis of disparity information in at least the secondcolumn of the disparity map 441.

For example, referring to FIGS. 44E and 44F, the processor 430 may amendinformation of at least one cell (e.g., 442-1) in the sixth column ofthe vertical disjoint set 442 on the basis of disparity information inthe sixth column of the disparity map 441. Further, the processor 430may amend information of at least one cell (e.g., 442-1) in the seventhcolumn of the vertical disjoint set 442 on the basis of disparityinformation in the seventh column of the disparity map 441.

FIGS. 44G and 44H illustrate an initial disjoint set 444, and a finaldisjoint set 445 generated from the initial disjoint set 444. The finaldisjoint set 445 may include segmented areas 445-1 and 445-2 of anobject, and a bordering cell or non-pattern area 445-2.

Obviously, the object segmentation method described above with referenceto FIGS. 44A to 44H is also applicable to a case where a cell isconfigured in units of 2×2 pixels.

FIGS. 45A to 45F are views depicting an object segmentation methodaccording to another embodiment of the disclosure.

FIGS. 45A to 45D illustrate disjoint sets 451, 452, 453, and 454according to an embodiment of the disclosure.

According to an embodiment of the disclosure, assuming that the size ofone cell is set to 2×2 pixels, the disjoint set 451 includes indexinformation for the top left pixels of the cells, the disjoint set 452includes index information for the top right pixels of the cells, thedisjoint set 453 includes index information for the bottom left pixelsof the cells, and the disjoint set 454 includes index information forthe bottom right pixels of the cells.

FIGS. 45E and 45F illustrate disjoint sets 455 and 455′ according to anembodiment of the disclosure,

According to an embodiment of the disclosure, the disjoint set 455 maybe formed on the basis of the index information of the disjoint sets451, 452, 453, and 454. The disjoint set 455 may include self-repaircells 455-1, and cells 455-2 that are merged in the same area. Thedisjoint set 455′ may be the result of completing object segmentation.

FIGS. 46A, 46B and 46C illustrate a triangulation method according to anembodiment of the disclosure.

Referring to FIG. 46A, the processor 430 may reconstruct 3D map points.For this purpose, the processor 430 may use an estimated shift amount,parameters of a lens system in the detector (or the camera and the lightreceiving unit), and calibration parameters.

The 3D map point reconstruction may be a process of calculating the X,Y, and Z values of each pixel in a captured cellular automaton patternby triangulation equations. The X, Y, and Z values may be calculated by[Equation 15].

$\begin{matrix}{\mspace{79mu} {{{Z = {b*{DETECTOR\_ WIDTH}\text{/}2\text{/}\sigma_{d}\text{/}{\tan ({FOVch})}}};}{{X = {{Z*\left( {1 - {2*\frac{Xp}{DETECTOR\_ WIDTH}}} \right)*{\tan ({FOVch})}} + b}};}{{Y = {Z*\left( {1 - {2*\frac{Yp}{DETECTOR\_ WIDTH}}} \right)*{\tan ({FOVcv})}}};}}} & \left\lbrack {{Equation}\mspace{14mu} 15} \right\rbrack\end{matrix}$

where b represents a baseline between the light projector and thedetector, σ_(d) represents the shift amount between an original patternand a captured cellular automaton pattern, and FOVch and FOVcv representfield or view values corresponding to angles of the detector, and Xp andYp represent cell coordinates in the captured cellular automatonpattern, related to a significant point of the projector.

FIG. 46B illustrates as set for Xp and Yp.

FIG. 46C illustrates a table 461 including depth information accordingto a Z value calculated by [Equation 15].

The process of projecting a cellular automaton pattern to a 3D space,and reconstructing 3D map points by using a captured cellular automatonpattern reflected from the 3D space has been described above in detailwith reference to various drawings.

Now, a description will be given of various embodiments of a process ofreconstructing 3D map points. The following embodiments and theforegoing embodiments for 3D map points may be implementedindependently, in combination, or supplementally.

FIG. 47 illustrates the types of cellular automaton lattices accordingto various embodiments of the disclosure.

FIG. 48 illustrates an example of cellular automaton rules for arectangular lattice.

Cellular automaton evolution will be described below in detail. Cellularautomata are implemented from disorderly initial states which arerandomly selected in such a manner that each site has a value of 0 or 1(generally, a probability of 1/2). The disorderly initial states may begeneral members of a set of all possible configurations. Patternsgenerated by cellular automata are acquired with some initial state. Thestructure of these patterns may be a representation of theself-organization of a cellular automaton.

A cellular automaton may begin with general initial configurations.Further, cellular automata include four types of automata evolutionaspects.

For example, cellular automata type 1 may evolve to synchronous finalstates. Cellular automata type 2 may produce disjoint periodicstructures. Cellular automata type 3 may exhibit a chaotic aspect, andproduce aperiodic patterns. A small change in initial states maygenerally lead to a changing area which linearly increases. Cellularautomata type 4 may exhibit complex local propagation structures.

Since the disclosure aims to generate a 2D pattern by coded spatiallocalization, 1D and 2D automata type 2 is of interest in thedisclosure. Further, automata of the type (2D cellular automata) take aquiescent state a few hours later.

Therefore, a predetermined 2D pattern may be generated by using[Equation 1] for 1D cellular automaton evolution and [Equation 2] for 2Dcellular automaton evolution (see FIGS. 49 and 50). In the case of 2Dcellular automata, [Equation 2] may be applied to a pattern generated by[Equation 1].

Cellular automaton evolution may be performed on the basis of a rulenumber. According to an embodiment of the disclosure, base 2 digits of arule number may determine cellular automaton evolution. The last bit ofthe rule number may specify the state of a cell when all neighbors areoff and the corresponding cell is also off. The bit next to the last bitmay specify the state of the cell when all of the neighbors of the cellare off and the corresponding cell is on.

Further, each pair of previous bits may specify what may occur when moreand more neighbors progressively (totalistic) become black. For example,bits 2 ⁰ and 2 ¹ may apply when none of four neighbor cells are on, 2 ²and 2 ³ may apply when one neighbor cell is on, 2 ⁴ and 2 ⁵ may applywhen two neighbor cells are on, 2 ⁶ and 2 ⁷ may apply when threeneighbor cells are on, and 2 ⁸ and 2 ⁹ may apply when four neighborcells are on. For example, rule 614 may correspond to bits{1,0,0,1,1,0,0,1,1,0}.

If an outer totalistic cellular automaton with 5 neighbors isconsidered, a subsequent state of a cell may be determined by the 4closest neighbors N, E, S and W, as well as the cell itself. In step 0to step 1, new states of 9 cells should be determined. Since a firstcell (at the top left corner) has an initial state of off, and allneighbors are off, bit 2 ⁰ is applied, thus indicating that a new stateis off. A second cell (in the middle of the uppermost row) has aninitial state of off, and one (a cell which is on from step 0) of itsneighbors is on. Since one neighbor is on, bits 2 ² and 2 ³ are applied.Since the initial state is off, bit 2 ² is applied, thus indicating thatthe new state is on. If the procedure jumps to a cell at the origin,then none of the neighbors are on, and the initial state is on. Thus,bit 2 ¹ is applied, thus indicating that the new state is on. Moredetails of rule 614 may be described. When an odd number of cells among5 cells are on, the state of a cell is switched to or maintained in anon state.

FIG. 51 illustrates cellular automaton evolution according to anembodiment of the disclosure. FIG. 51 illustrates the initial five stepsamong evolution 0 to evolution 4 generated by rule 614.

FIG. 52 illustrates a 2D cellular automaton rule according to anembodiment of the disclosure. Cellular automata are related to rule set#736 according to Wolfram terms in a book entitled “New Kind ofScience”.

Cellular automata were introduced by J. von Neumann to model physicaland biological phenomena, in particular, for von Neumann's pioneeringstudies of self-reproduction. A cellular automaton is a special kind ofautomaton that includes a large number of mutually connected simpleidentical components, determined on an n-order lattice of cells within adrawing, which has a finite set of possible values. A cellular automatonevolves in discrete time steps, and a value (local state) taken by aspecific cell is affected by the cell values of its neighbors in aprevious time step according to a function δ known as a cellularautomaton in the drawing.

A global state is defined as a vector of local states for all cellularautomata during cellular automation.

A cellular automaton A will be introduced by a graph G which is a set ofnodes and a set of edges E. An edge e∈E is an unordered pair of nodes,{v, v′}∈V. The neighborhood of a node is the set of nodes directlyconnected to it in the graph by an edge. Further, v′=nbhd(v), and if {v,v′} are included in V, this means that v′ is a neighbor of v.

Herein, a deterministic finite cellular automaton A is a finite set ofstates Q, a finite set of inputs, X, and a transition function δ: QxX→Q.Cellular automata are a finite or infinite network of identicaldeterministic finite state automata and a graph structure such that:

-   -   1. each node has the same number (finite number) of neighbors;    -   2. at each node, we have a fixed order of the neighbor nodes;        and    -   3. the next state of the automaton at node v is always the same        function δ of its current state and the current states of its        neighbors. Thus, the ordered list of states at neighbor nodes is        the input to the automaton at node v.

The ordered list of the states of the neighbors of node v is denoted byq_(wEnbhd)(v). It is assumed that a cellular automaton of the disclosureis implemented as a set of pixels with integer coordinates in a 2D IRimage including neighbors in relation to, for example, the von Neumannor Moor neighborhood in some N-dimensional Euclidean space. Further, ifthe automaton at node v is in state q₀ and all of its neighbors are instate q₀, the automaton is in the quiescent state q₀∈Q so that theautomaton at node v will still be in state q₀ in the next time step of asynchronous mode. An automaton configuration is any assignment of localstate values to the set of automata at nodes in a finite subgraph of G.

Usually, a cellular automaton is used to update the state of the finiteautomata at all of its nodes simultaneously (synchronous mode) and indiscrete steps (evolution epochs). Therefore, for all epochs t>0, if atepoch t, each node v is in some state q_(v)(t), node v is in its nextstate q_(v)(t+1) in the next epoch t+1. The next state is given by

q _(v)(t+1)=δ(q _(v)(t), q _(w∈nbhd(v)))

q _(v)(t+1)=δ(q _(v)(t), q _(W) _(∈nbhd(v)) )

Therefore, the new state of the automaton at node v is given by thelocal update rule as a function of q_(v)(t), the current state of nodev, and the finite list of all current states of all nodes in theneighborhood of node v. The global state of the cellular automaton A ineach epoch includes the states q_(v)(t) of its all nodes in each epoch.That is, an update in the global state is performed in the synchronousmode. Herein, the updates of the local component automata are notrequired to take place synchronously. Further, if each one of theupdates of the local component automata is updated to its next state anunlimited number of times as (locally discrete) time goes on, anasynchronous automata network may be spoken of.

Following the general method and theorem of Nehaniv for each synchronouscellular automaton A on the graph another cellular automaton A′ isconstructed on the same graph. If each node v has 3 n² states and theasynchronous update method is applied to each node, that would beasynchronous. Therefore, according to the theorem, the global state of Ais completely determined by a spatial-temporal section of the behaviorof A′. This mathematical theorem implies that all computations that canbe carried out on any synchronous automata network can be recovered fromthe computation of an asynchronous automata network with no constraintson how the updates actually occur later.

The local automata of the asynchronous cellular automaton A′ may beconstructed from local automata of the synchronous cellular automaton.It is assumed that a local automaton of A has states Q={q₀, . . .,q_(n−)} with q₀ quiescent and an update function δ: QxX→Q. The statesof a local automaton of A′ are 3 n² states, QxQxr where r={0, 1, 2}. Allneighbors are ready when none of the neighbors are in a third state (0).It is assumed that node v is in a state (q, q′, r) and has aneighborhood in a state (q_(w), q′_(w), r_(w)).

Then, if r=0, the next state of the node is

$\begin{matrix}{{\delta^{\prime}\left( {\left( {q,q^{\prime},r} \right),\left( {q_{w},q_{w}^{\prime},r_{w}} \right)} \right)} = \left\{ \begin{matrix}\left( {{\delta \left( {q,q_{w}^{''}} \right)},q,1} \right) & {{ifneighbors}\mspace{14mu} {are}\mspace{14mu} {ready}\mspace{14mu} (0)} \\\left( {q,q^{\prime},0} \right) & {otherwise}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 17} \right\rbrack \\{q_{w}^{''} = \left\{ {{\begin{matrix}{q_{w},} & {{if}\mspace{14mu} {neighbor}\mspace{14mu} {node}\mspace{14mu} w\mspace{14mu} {is}\mspace{14mu} {in}\mspace{14mu} {state}\mspace{14mu} {of}\mspace{14mu} \left( {q_{w},q_{w}^{\prime},0} \right)} \\{q_{w}^{\prime},} & {{if}\mspace{14mu} {neighbor}\mspace{14mu} {node}\mspace{14mu} w\mspace{14mu} {is}\mspace{14mu} {in}\mspace{14mu} {state}\mspace{14mu} {of}\mspace{14mu} \left( {q_{w},q_{w}^{\prime},1} \right)}\end{matrix}\mspace{20mu} {If}\mspace{14mu} r\mspace{14mu} {is}\mspace{14mu} \left\{ {1,2} \right\}},{{the}\mspace{14mu} {next}\mspace{14mu} {state}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {node}\mspace{14mu} {is}}} \right.} & \left\lbrack {{Equation}\mspace{14mu} 18} \right\rbrack \\{{\delta^{\prime}\left( {\left( {q,q^{\prime},r} \right),\left( {q_{w},q_{w}^{\prime},r_{w}} \right)} \right)} = \left\{ \begin{matrix}\left( {q,q^{\prime},{r + {1\mspace{11mu} {mod}\mspace{14mu} 3}}} \right) & {{if}\mspace{14mu} {neighbors}\mspace{14mu} {are}\mspace{14mu} {ready}\mspace{14mu} (r)} \\\left( {q,q^{\prime},r} \right) & {otherwise}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 19} \right\rbrack\end{matrix}$

In general, the self-reproduction property of a cellular automaton isthe possibility of generating a copy of the same pattern for apredetermined configuration. This cellular automata property enablesconstruction of scheduled algorithms in a recognition operation. In thecase of a synchronous update of a cellular automaton and image size, M×Npixels, the same number of cellular automata may be implemented in orderto recognize a part (or a current posture) of a whole pattern. In thecase of an asynchronous update of a cellular automaton, recognition isperformed once per current position.

FIG. 53 illustrates cellular automata according to an embodiment of thedisclosure.

In FIG. 53, cellular self-reproduction may proceed in the order ofcellular automaton finite states (spatial code) 531, cellular automatonevolution 532, and cellular automaton generic pattern 533 after a fewevolutions.

According to the forgoing asynchronous mode, self-reproduction is theevolution of a few copies of a given automaton. Such evolution processestake place simultaneously, instead of in the synchronous mode.Accordingly, in each pixel of a captured image with a reflected pattern,cellular automaton evolution begins with an initial spatial code whichcontinuously changes until the number of image changes becomes minimal.The captured reflected pattern is an image, and a change of the imagetakes place along with local automaton evolution, if a pixel value doesnot correspond to a pixel of the pattern.

FIGS. 54A to 54D illustrate cellular automata according to anotherembodiment of the disclosure.

FIGS. 54A to 54D illustrate asynchronous self-reproduction for a fewcopies of a cellular automaton in different positions of a lattice.

Specifically, FIG. 54A illustrates initial states of a cellularautomaton. Different initial states may be coded spatial information onthe cellular automaton lattice. FIG. 54B illustrates a cellularautomaton after N-step cellular automaton evolution. FIG. 54Cillustrates a cellular automaton after cellular automaton evolution in nsteps (n>N). FIG. 54D illustrates a cellular automaton after cellularautomaton evolution in n steps (n>>N).

A reflected image of a projected pattern (a reflected pattern) may becaptured with partial loss. Thus, cellular automaton self-recovery isrecovering an image (strips, spots, or dots) of a lost pattern from alow-reflective (black and transparent) surface. An input to a recoveryalgorithm is a reflected pattern with a lost part or a part with noise.Herein, an initialized cellular automaton is changed until the wholepattern is recovered. Self-recovery steps are applied to the result ofself-reproduction for disparity map enhancement.

FIGS. 55A to 55D illustrate self-recovery of a cellular automatonaccording to various embodiments of the disclosure.

In FIG. 55A, a dark part represents the lost part of a pattern. In FIG.55B, the pattern is lost in a dark area of a ball. FIG. 55C illustratesrecovery after N evolutions of a cell automaton. FIG. 55D illustrates awholly recovered pattern.

Generation of a genetic pattern by 1D cellular automata includes loadinginitial states and rules from a memory of a mobile device. In relationto 1D cellular automata, initial states may be represented as rows orcolumns of an image (or a lattice of the same size) including a sequenceof values randomly generated for all species of cellular automata.

The disclosure proposes generation of a genetic pattern from one pointon a 2D lattice of the automaton through 1D automaton evolution asillustrated in FIGS. 55A to 55D. A basic cell of a lattice may beselected to have one pixel or a larger width according to the opticallimitations of a projection and capturing assembly, including a powerconsumption limitation. An evolution process for growth of a geneticpattern continues until epoch t⁰ by [Equation 1], which describes theautomaton rule. Although epoch t⁰ may be selected independently, it isthe states of cellular automata when the states of each species in everystate have slowly distributed values along a full lattice size in adirection perpendicular to an epipolar line. The epoch t⁰ depends on theorientation of a baseline between an IR camera and an IR projector,

FIG. 56 illustrates generation of initial states for a cellularautomaton according to an embodiment of the disclosure.

In FIG. 56, as the lattice of a pattern rotates, a baseline is directedvertically. The size of the lattice is 640×480. The epipolar line isdirected along the width of the pattern, that is, the rows of an IRimage.

All epochs that evolved from epoch t⁰-t^(width) are related to a zerodisparity Z_(M) from [Equation 7].

$\begin{matrix}{\sigma_{Z} \sim \frac{\sigma_{d}}{b\text{/}h}} & \left\lbrack {{Equation}\mspace{14mu} 20} \right\rbrack\end{matrix}$

where σ_(z) is a measurement accuracy, b/h is a proportionalcoefficient, and σ_(d) is the accuracy of disparity measurement. Anepoch may be performed within a 3D correction pair of an IR camera andan IR projector. Further, an epoch may be adjusted by using the originof the IR camera, Z_(M)=0. It should be noted that each epoch isrepresented as a column or S^(f) (see the drawing) columns in an IRimage with respect to the orientation of the baseline.

FIGS. 57A and 57B illustrate patterns generated in an initial stageaccording to an embodiment of the disclosure.

Specifically, FIG. 57A illustrates a pattern generated until an initialepoch by a 1D cellular automaton. FIG. 57B illustrates rough shifting ofa rectangular area in an IR image when an object is positioned within ascene. In FIG. 57B, a cellular automaton lattice includes cells of640×480 image pixels. A baseline is directed vertically.

Generation of a genetic pattern by 2D cellular automata is similar tothe case of applying 1D automata. The difference lies in the number ofdimensions and the initial state generation method. Generation ofinitial states includes generation of a 2D pattern based on 1D cellularautomata from one point, as described with reference to FIG. 56. Thismethod evolves a 2D cellular automaton by using 2D (5 or 9) neighbors ona 2D lattice with S^(f)×S^(f) rectangular cells by [Equation 2]. Asdescribed above, the evolution proceeds until the quiescent state of a2D cellular automaton of cellular automata type 2 is detected.

FIG. 58 illustrates a genetic pattern according to various embodimentsof the disclosure.

FIG. 58, a genetic pattern generated from a genetic pattern generated by1D cellular automata by applying 2D cellular automata on a 2D 640×480lattice is shown.

A pattern generated by [Equation 1] or [Equation 2] is a referencepattern, which is used to construct a transparent binary mask for an IRprojector, including an improvement means such as amplitude correction.In embodiments of 2D cellular automata, an initial state pattern isgenerated by 1D automata, and is a support pattern for a zero disparity.

A genetic pattern described in the disclosure is different fromconventional patterns based on a pseudo-random non-correlationsemi-period and properties based on the above description. When agenetic pattern generated by 2D cellular automata satisfies the periodicgrid condition of mini-patterns due to the use of self-organization ofcellular automata. (U.S. Pat. No. 8,090,194 B2, “3D Geometric Modelingand Motion Capture Using Both Single and Dual Imaging” E. Gott:don,Mantis Vision Ltd( Aug. 8, 2007)), a genetic pattern generated by 1Dcellular automata satisfies the mandatory conditions of a non-correlatedpattern, and may be used by the methods disclosed in U.S. Pat. No.8,090,194 B2.

FIGS. 59A and 59B illustrate matched areas according to variousembodiments of the disclosure.

FIGS. 59A and 59B illustrate matched areas 171 and 172 used to detect anoffset between a reference pattern and a reflected pattern.

In the method according to the claims of U.S. Pat. No. 8,090,194 B2, acaptured image is processed, an offset between a projected patterncaptured once during correction and each of the patterns in a pluralityof areas (a sliding window scheme, or the like) captured from an imagehaving a reference pattern is detected, and the distance to each of theareas is determined according to each offset. In this case, generateddisparity maps may be similar to initially disclosed ones except for aused improvement such as a duty cycle gray code and N-ary modulation.Herein, a matching process may be performed by convolution of the areasfrom a reflected reference pattern. In contrast to the fact that agenetic pattern is generated without n-ary modulation, a spatial codemay be decoded by a Fourier transform interpretation in the frequencydomain. Therefore, geometrical primitives self-organized in the geneticpattern have distinguishable frequency peaks.

In another aspect, a 2D cellular automaton of cellular automata type 2has continuous mini-patterns distinguishable on a periodic grid.Therefore, the method according to the claims of U.S. Pat. No. 8,150,142B2 (“Depth Mapping Using Projected Patterns”, B. Freedman, Prime SenseLtd. (Apr. 3, 2012)) applies demodulation to points in an image gridusing correlations with a reference pattern, as follows.

$\begin{matrix}\begin{matrix}{- 1} & {- 1} & {- 1} & 0 & 1 & 1 & 1 \\{- 1} & {- 1} & {- 1} & 0 & 1 & 1 & 1 \\{- 1} & {- 1} & {- 1} & 0 & 1 & 1 & 1 \\{\frac{1}{9}\mspace{11mu} 0} & 0 & 0 & 0 & 0 & 0 & 0 \\1 & 1 & 1 & 0 & {- 1} & {- 1} & {- 1} \\1 & 1 & 1 & 0 & {- 1} & {- 1} & {- 1} \\1 & 1 & 1 & 0 & {- 1} & {- 1} & {- 1}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 21} \right\rbrack\end{matrix}$

[Equation 21] may be a combination for a correlation operation and localarea averaging for a reflected pattern. Correlation described in U.S.Pat. No. 8,150,142 B2 is selected due to features of the result ofmodulating a binary background and may be calculated by [Equation 22],instead of the convolution of each pixel in an IR image.

correlation of point_(i,j) =1 _(−2,2) +a _(2,−2) −a _(2,2) −a _(−2,−2)  [Equation 22]

where a_(−2,2) a_(−2,2) is a local average around a pixel (i−2, j+2).Thus, the convolution between this matrix and the pattern will providelimit values at points of the grid. With the use of a genetic patternbased on a rule, for example, “rules #736”, and correlation ([Equation21])-based derivation, sub-pixel grid points may be detected by asimilar correlation function along with modulation based on a patternstructure.

FIG. 60 illustrates a mapping relationship between a pattern image and acellular automaton lattice according to an embodiment of the disclosure.

It is assumed that a 2D cellular automaton is defined on a rectangularlattice, basic cells of the cellular automaton are an area of sizeS_(f)×S_(f), and a pair of two neighbor cells is of size S_(f)×S_(L)satisfying the following.

2S_(f)≥S_(L)   [Equation 23]

Thus, if c=1/4(S_(f)+S_(L)), then for each cell of the automaton on thelattice the following is given.

cell_(i,j) =a _(0,0) +a _(c,0) +a _(−c,0) +a _(0,c) +a _(0,−c) −a_(−c,c) −a _(c,−c) −a _(c,c) −a _(−c,−c)   [Equation 24]

Herein, limit points of this function across an IR image are thepositions of sub-pixels of points on the illustrated grid.

FIGS. 61A and 61B illustrate mapping between a computed grid and aphysical grid in a cellular automaton lattice according to an embodimentof the disclosure.

Referring to FIGS. 61A and 61B, after sub-pixels on a physical grid 191in an IR image are localized, a reflected pattern is mapped to a generallattice 192 of a computed plane. Additional mapping of correlationsbetween a reference pattern and a mapped and reflected pattern areperformed by convolution in a sliding window scheme.

In the disclosure, the mapping is performed by cellular automatonevolution, and rules are detected by [Equation 24]. It is assumed thatlocal average components are replaced with non-weighted sums used tobuild a rule as follows.

$\begin{matrix}\begin{matrix}a_{{- c},{- c}} & a_{0,{- c}} & a_{c,{- c}} \\a_{{- c},0} & a_{0,0} & a_{c,0} \\a_{{- c},c} & a_{0,c} & a_{c,c}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 25} \right\rbrack\end{matrix}$

The structure of the rules from [Equation 25] depends on the globalaspect of the genetic pattern. For this reason, the designs of the rulesare individually made in each quiescent state of a 2D automaton used ingenerating a genetic pattern.

According to an embodiment of the disclosure, multiple sums of pixelvalues in the respective cells of a captured IR image may be used inorder to map a reflected pattern onto a sub-pixel calculation plane.Further, cellular automata rules may be used for an evolution process.Further, convolution may be performed to improve matching relations.Accordingly, transposed sum tables and approximated algorithms reducepower consumption. The intensities of an IR image may be pre-calculatedeffectively for sub-pixel disparity matching in cellular automata by thefollowing ‘method of approximating a multimedia signal filteringalgorithm’.

Meanwhile, a multimedia signal is a 1D or higher-dimensional array ofvalues preserved in information within a digital representation, and theposition of a value corresponds to one piece of unique information. Forexample, an image is a 2D array of pixels where a pixel corresponds to aj^(th) row and i^(th) an column. Sound is a 1D array of phonemes where aphoneme corresponds to an i^(th) time point. In addition, each part of amultimedia signal has spatial relationships with its neighboring parts.The relationships depend on a grid of the multimedia signal as theneighborhood of the part. For example, an image is defined on a 2Drectangular grid. Since nodes are positioned at pixels, each pixel has 8neighboring pixels. A phoneme has 2 neighboring values and itself. In awide domain of a computer vision, many multimedia signal processingalgorithms convert an original signal according to the values ofneighbors and the value of a current part. For example, the multimediasignal processing algorithms include image or sound filtering algorithms(gradient, image blurring, sound modulation, noise cancellation, andbilateral filtering). The enumerated basic processing algorithms are apart of more complex algorithms such as AR/VR content generation, objectrecognition, classification, and localization, which may be accompaniedby executions of basic algorithms repeated a few billion times. Further,the performance of the basic algorithms involves the read/write of thevalues of a large number of neighbors (memory access) and theirmanipulations (accumulations and multiplications). Convolution is themathematical operator used most f©r the foregoing computations forneighbors. In general, neighbor relations are represented as an array ofvalues indicating measurements (estimates or coefficients) of theinfluence of signals on a current part. In general, each measurement inan array is referred to as a weight, and an array of weights is referredto as a kernel (see FIG. 62). That is, the mathematical concept ofconvolution is referred to as the weighted sum of neighbors.

outputting signal_(i,j=Σ) _(K=0) _(MN) _(a) _(s) _(signal) _(k)neighborhood of piece in 9i,j pose =  [Equation 26]

where M and N are the width and height of a 2D kernel. For example, M×1is the length of a 1D kernel.

FIG. 62 illustrates convolution according to an embodiment of thedisclosure.

FIG. 62 illustrates weights 1/9 of a 3×3 array (2D kernel) andconvolution of a 2D multimedia signal (image). The result of convolutionis a new output image in which each pixel value is the weighted sum ofneighbors of the pixel in an original image, which provides digitalconnection pictures or photos called mini-patterns, which are formedwith weights through the connectivity, density, or concentration of thesame values.

The weights of a kernel are variables which are dependent on effectsexpected from a resulting signal and job. However, if an accurate kernelfor a specific job is considered, it is constant during programexecution. For example, the kernel illustrated in FIG. 40 is dedicatedto blurring an image. Some kernels are constructed from the priorknowledge of a user, and other kernels are iterative or detected byneural networks of convolution and machine learning schemes such asAda-boost and SVM. All weights within a kernel may construct somespatial connection pictures or photos called mini-patterns.

The ‘method of approximating a multimedia signal filtering algorithm’discloses the approximated performance of convolution. The essentialidea of the method is that multiplication is split into summations byusing the spatial structure of kernels as a set of mini-patterns withinthe kernel. Such splitting is available by the spatial connectionconfigurations such as the mini-patterns. Further, the number ofmini-patterns in a kernel matches K approximated elements, and the sumis a linear combination of the following form approximate to [Equation13].

outputting signal_(i,j)=Σ_(k=0) ^(K) a _(k)mini_pattern_(k) ^(approx)  [Equation 27]

Since mini-patterns are comprised of kernel M×N parts (weights), K≤M×Nfor one kernel. If a large number of kernels are applied to the samemultimedia signal, such approximation based on grouping of weights intomini-patterns may be useful. Herein, since an algorithm includes kkernels, it is assumed that k linear combinations may be acquired forevery kernel.

outputting signal_(i,j)=Σ_(l=0) ^(k)Σ_(m=0) ^(K) a_(m,l)mini_pattern_(m) ^(approx)   [Equation 28]

Herein, a mini-pattern is merely the sum of parts of a multimediasignal, and the form of mini-patterns may depend on a kernel. This meansthat multipliers in all K×K forms, mini_pattern_(m) ^(approx) arecombinations of all possible forms for given kernels k×M×N. Herein, ifall weights of the mini-patterns are equal, multipliers a_(m,1)=a₁ maybe introduced to a squared sum ([Equation 29]) which is the sum of allparts of a mini-pattern in an original signal. The sum is a part of thewhole kernel ([Equation 30]), and a partial sum.

outputting signal_(i,j)=Σ_(l=0) ^(k) a _(l)Σ_(m=0) ^(K)mini_pattern_(m)^(approx)   [Equation 29]

outputting signal_(i,j)=Σ_(m=0) ^(K)mini_pattern_(m) ^(approx)  [Equation 30]

Assuming that a table of mini-kernels in all possible M forms isconstructed, the table will have M memos. A new image constructed withpartial sums ([Equation 30]) of a specific mini-pattern for each of theposition of parts of a multimedia signal Row×Col is placed in each memoof the table.

Partial sum image=Σ_(j=0) ^(Row)Σ_(i=0) ^(Col)Σ_(m=0)^(K)mini_pattern_(m) ^(approx)   [Equation 31]

An image related to the above description is referred to as a partialsum image, and a table related to the above description is referred toas a partial sum table. Additionally, the form of a mini-pattern sum ofthe partial sum image may include an in-kernel offset. Therefore,mini-patterns, for example, 2×1 positioned in two different kernels withdifferent offsets, may use the same partial sum image in the table.Then, a different a_(I) may be multiplied once every kernel, instead ofm multiplications of a_(m,l).

When more and more kernels are used, and thus K<<k×M×N, themini-patterns are repeated in a few kernels, which brings aboutadditional advantages. More ones have 40% zeros or more across mostkernels. For example, if a convolution neural network for classificationbrings an 80% reduction of power consumption for rectangular 2Dpatterns, the network may have 100 billion kernels, and a tenfoldacceleration rate. When possible mini-pattern forms are known and thenumber of the mini-pattern forms is finite and small, a unique advantageis achieved. This case takes place when a multimedia signal has binaryvalues 0/1, −1/1, or 1, 2, 4 . . . 9. An example of such an embodimentis a genetic pattern after binarization. Herein, mini-patterns areknown, self-organized genetic primitives are finite, and their forms areless variable.

Further, if a plurality of kernels is ideally convolved with the samemultimedia signal (for example, one image) as described before, theresults are accumulated into a new single image. A set of the kernels isreferred to as a layer. The convolution takes place in machinelearning-based algorithms for object recognition, speech recognition,handwriting recognition, classification, or localization. Theconvolution is often used in pyramid matching algorithms forinterpolation and location tracking. The creativity of such a case hasnothing to do with the convolving order of kernels. The kernels may beselected randomly. For this purpose, the ‘method of approximating amultimedia signal filtering algorithm’ is disclosed for combining allkernels to another with dimension D+1 where D is the dimension ofkernels. For example, 2D kernels are combined into one 3D kernel, andoptimum mini-patterns are also 3D, not 2D as illustrated in FIG. 21.Therefore, the mini-patterns are formed with the volumes of parts of aninput multimedia signal. Since mini-patterns fewer than K may beobtained in this manner, the mini-patterns may have significantadvantages in terms of power and the number of computations, asillustrated in FIG. 22. In the case of a genetic panel of a structuredlight camera, a binary representation is used, and thus the 3D arraywill be very scarce.

The essential idea of a convolution-based approach is that each part ofa genetic pattern is represented as a plurality of 2D binary kernels.Thus, a disclosed algorithm splits the product-sum operations ofconvolution in order to separate a sum and a product from each other ina standard case. Further, if the binary representation of a pattern isused, multiplication is emphasized in embodiments with a geneticpattern. The splitting reduces power consumption and computationcomplexity in a matching process for 3D environment reconstruction. Inorder to solve the matching, the method performs approximation,calculation of a table of partial sums, and weighting.

Kernel approximation of a genetic pattern handles each window of adefined kernel size by using the spatial structure of a kernel G²(x, y)of each actual value within a window, so as to reconfigure it by usingthe same linear combination of partial sums ([Equation 27]) within a 2Darea having approximately the same values within a kernel(R^(approx)=mini_pattern_(m) ^(approx)). If intensities of an IR imageare used, a normalization item should be introduced. Therefore, thepartial sums of mini-patterns have semi-equal values inside them due toan image of actual values (noise, pattern modulation, and the dependencyof intensity on depth). If O is the accuracy of kernel approximation, anoutput signal is given by

outputting signal_(i,j)≅Σ_(l=0) ^(k)Σ_(m=0) ^(K) a_(m,l)mini_pattern_(m) ^(approx)+0(θ)   [Equation 32]

For example, FIG. 63A provides a brief description of approximation inthe case of actual values, and FIG. 6313 provides a brief description ofapproximation in a binary case. FIGS. 63A and 63B will be describedbelow.

FIGS. 63A and 63B illustrate the concept of kernel approximationaccording to an embodiment of the disclosure.

FIG. 63A illustrates a 3×3 kernel of actual values which areapproximated with three approximation areas and three weights accordingto a spatial density map of connected components, and FIG. 63Billustrates a case in which a binary representation of a referencegenetic pattern is given, where each approximation kernel within awindow shifting all cells by 1 cell is approximated with k connectedcomponents (k=2 (left), and k=3 (right)).

If a set of optimum areas (k<<M×N) and intensity values are used,reconstruction of the same linear combination of the respective kernelsof a genetic pattern amounts to determining their weights. Further, ifbinary values are used, all k components connected within a windowapproximate k areas R^(approx).

In embodiments using a genetic pattern of kernels having M and N, eachranging from 2 to 13, an exhaustive search or a BFS algorithm, whichminimizes the average squared error of both sides in [Equation 20], isused. Therefore, an approximation algorithm covers all combinationsavailable for a kernel size. The search is discontinued when there is noarea R^(approx) which minimizes the difference from an original kernelor from which an approximation accuracy is achieved. A set detected forlinear combinations of partial sums for all windows is used for theapproximated performance of matching algorithms. The originalperformance of matching convolution A is comprised of a large number ofexhaustive convolution computations as illustrated, and approximatedperformance A′ performs light accumulation computations with two offsetsof important values for each area R^(approx) equal to an actual value aand a table z,41 of partial sums. A known image size is used topre-calculate offsets for sets of partial sum images for all k areas.These images are accumulated in one partial sum table denoted by

.

After the transpose calculation, convolution calculates a weightedlinear combination in the following steps.

1. Partial sum calculation (accumulation computations).

2. Weighting (multiplications)

3. Accumulation of weighted sums

FIG. 64 illustrates convolution according to an embodiment of thedisclosure.

Herein, for convolution, multiplication-accumulation steps (top) andseparate multiplication and accumulation (bottom) are used.

In general, a partial sum table includes a set of 2D arrayscorresponding to approximated mini-patterns in a window within a geneticpattern. These arrays are images of sums for configuring each area suchas 1×, 1×2, . . . M×M or more complex ones according to the connectedcomponents. The sum of an input image at each position of such an imagecorresponds to a mini-pattern, as illustrated in FIG. 26.

Weighting may be the simple multiplication of one of the images from thetable by a constant a according to pre-calculated offsets for imageswithin the table

. Weighted partial sums corresponding to outputs may be accumulated. Ina binary genetic pattern, all weight coefficients are 1. Therefore, theresult is one-time accumulation.

According to the asynchronous mode introduced by [Equation 15] to[Equation 17], self-reproduction is simultaneously evolving a pluralityof copies of a given automaton from a support pattern in an initialstate according to a given rule.

Multi-dimensional cellular automata for which each rule of a set isrepresentable as a kernel having the size of a captured image of 3×3cells or 3S_(f)×2S_(f) pixels, as well as 2D cellular automata, areconsidered. Herein, a cell value is calculated by using accumulatedintensities of all pixels in S_(f)×2S_(f). Therefore, the update statefunction of an external totalistic cellular automaton ([Equation 13])may be approximated through a linear combination of connected componentsfor the following representation of actual values of a pattern([Equation 27]).

a _(ij) ^(t+1)=(a _(i−1,j) ^(t) +a _(i,j) ^(t) +a _(i+1,j) ^(t) +a_(i,j−) ^(t) +a _(i,j+1) ^(t))≅Σ_(k=0) ^(K) a _(k) R _(k) ^(approx)+0(θ)  [Equation 33]

Further, the approximation error is 0 (σ=0) in a binary representationgiven as the following equation.

a _(ij) ^(t+1)=(a _(i−1,j) ^(t) +a _(i,j) ^(t) +a _(i+1,j) ^(t) +a_(i+1,j) ^(t) +a _(i,j−1) ^(t) +a _(i,j+1) ^(t))=Σ_(k=0) ^(K) R _(k)^(approx)   [Equation 34]

Each approximation area is a connected component in the neighborhood ofa cell in a position (i, j) on a cellular automaton lattice.

$\begin{matrix}{R_{k}^{approx} = \left\{ \begin{matrix}{1,} & {{{if}\mspace{14mu} x{\left\lbrack {{i - 1},{i + 1}} \right\rbrack}},{y{\left\lbrack {{j - 1},{j + 1}} \right\rbrack}\mspace{14mu} {and}\mspace{14mu} C_{i,j}R_{k}^{approx}}} \\{0,} & {otherwise}\end{matrix} \right.} & \left\lbrack {{Equation}\mspace{14mu} 35} \right\rbrack\end{matrix}$

According to an embodiment of the disclosure, a captured IR image isfirst converted to a partial sum table

, and then a cellular automaton is evolved by using the approximationrules described in [Equation 34] and [Equation 35] on a partial sumimage corresponding to connected components of an automaton rule.

Based on the rules ([Equation 26]), an introduced 2D cellular automatonmay be approximated in order to efficiently map a physical sub-pixelgrid in the captured image to a grid calculated as follows.

$\begin{matrix}\begin{matrix}{a_{ij}^{t + 1} = \left( {a_{{i - 1},j}^{t} + a_{i,j}^{t} + a_{{i + 1},j}^{t} + a_{i,{j - 1}}^{t} + a_{i,{j + 1}}^{t} +} \right.} \\\left. {a_{{i + 1},{j + 1}}^{t} + a_{{i - 1},{j + 1}}^{t} + a_{{i + 1},{j - 1}}^{t} + a_{{i - 1},{j - 1}}^{t}} \right) \\{= {\sum\limits_{k = 0}^{K}R_{k}^{approx}}}\end{matrix} & \left\lbrack {{Equation}\mspace{14mu} 36} \right\rbrack\end{matrix}$

Therefore, in an embodiment of the disclosure, a partial sum table

is calculated and then grid mapping is approximated, thereby supportingsub-pixel localization prior to self-reproduction for correspondencematching or pixel-wise convolution.

An IR camera captures a reflected genetic pattern, a 2D image obtainedfrom the captured genetic pattern is interpreted, and then theinterpreted image is processed by a program in the processor of a mobiledevice. The program may use a triangulation scheme of reconstructingdepth information from a disparity map of the correspondencerelationship between matched areas of the captured reflected pattern anda generated reference pattern. Disparity (mismatch) occurs due to adisparity effect in which the same object regarded as positioned indifferent depths and having different views is shifted along an epipolardirection. The shift does not exist along a perpendicular directionafter calibration of a 3D camera by a depth measurement. For example,the shift is performed in a row-wise direction, and does not exist alonga column-wise direction.

Therefore, the same parts of patterns need to be matched along the rowdirection. The matched areas are determined by deciphering a spatialcode. The deciphering process includes a step of taking a part of acaptured pattern and performing cellular automaton evolution from thepart until each new evolution epoch brings proper matching betweenreflected and evolved ones. Along the epipolar direction, the part maybe an area in a j^(th) row and an i^(th) column or a subpart of the areataken in a perpendicular direction in a 1D case, and an area of any formin a 2D or higher-dimensional case. The measurement of proper matchingis the Hamming distance between two binary codes that have spatiallengths. The Hamming distance may be generated by using cellularautomaton evolution of the binary codes and is calculated from thereflected pattern of a captured image.

According to an embodiment of the disclosure, the multi-dimensionalcellular automaton self-reproduction property may be used in a moregeneral context, instead of an irreversibility restriction. A scene ofan environment having only one rectangular object positioned on a flattable illustrated in FIG. 58 may be captured. Then, the rectangular areaof the object in an IF. image is shifted along an epipolar line. Aspatial code in the position of each cell is deciphered in a 2D pattern.Cellular automaton evolution is performed on all columns of thereflected pattern in a direction perpendicular to the epipolardirection. For every row in epoch t⁰ to t^(max), minimum shift discoveryis performed. Therefore, the disparity is given by shift=d−i where t^(d)is an epoch and i represents a current column, when the Hammingdifference between a reference pattern and a reflected pattern in aj_(th) row is minimal for an i^(th) column. Herein, the uniqueness ofeach epoch is used. In this example, every row related to the surface ofthe table has the same shift (disparity) t^(table)+σ^(noise) along theepipolar line, where σ^(noise) is 2D is a table depth. Additionally, thealgorithm removes obscurity in the next steps by displaying edge andshadow cells. Such a disparity map may include areas with non-uniformdepths and mismatched areas, such as holes and artifacts. Areas havingdepths may be used in reconstructing or segmenting (splitting) a scenethat has pixel-level accuracy.

To detect a more accurate disparity estimate, the program forms aninitially estimated new pattern form in columns and rows in each area.This construction process includes an operation of shifting back eachmatched area to a zero disparity position according to disparityestimates, and thus may be related to a reference genetic pattern. Thedetected new pattern may include holes in non-matched or obscure cells.After the program takes the constructed pattern, the program applies theself-recovery process from it. Self-recovery involves global cellularautomaton evolution which means evolution from the constructed pattern.An enhancement and recovery process may be performed in the asynchronousupdating mode according to [Equation 15] to [Equation 17]. Herein,pattern generation and initial estimation are applied in theasynchronous mode by [Equation 3]. The global evolution is intended tominimize the difference between the reference pattern and theconstructed pattern. Accordingly, evolution in each cell asynchronouslyrecovers insufficient or lost information such as holes or artifacts.This enhancement is made possible by global limitations provided by acomplete reference genetic pattern.

After the constructed pattern is recovered, the disparity value of amatched area is allocated to the recovered cells of the pattern. Then,all disparities are incorporated into one disparity map having theresolution of a cellular automaton lattice, that is, the resolution of acell of S_(f)×S_(f) pixels or the resolution of a pixel. Sub-pixelvalues are obtained by the backward mapping of grids to the physicalplane of the IR image. Therefore, a sub-pixel disparity map includingforward mapping errors or other distortions described above is acquired.To emphasize the errors, the error optimization problem may be solved byLevenberg-Marquardt non-linear optimization for static correction asalready used in the disclosure in order to adjust all sub-pixel values(texture or color when an ION or IrW-RGB matrix of an IR camera is usedin the embodiments) according to the constraint of non-continuity.

A genetic pattern generated by means of a 2D cellular automaton isdeciphered in a very similar manner to the 1D cellular automaton case.The difference lies in that the afore-described reference patterngeneration and self-reproduction are performed in an area, instead of inrows (columns). Therefore, the program generates a supporting 2D patternby the evolution of a 1D cellular automaton (for example, the same 1Dcellular automaton as described with reference to the drawings) in theinitial steps. This pattern is used as a support for a zero disparity.In this manner, the program uses various shifts of the support in orderto initialize the self-reproduction of the 2D cellular automaton foreach cell and its neighborhood. Therefore, the program detects aninitial disparity estimate. Then, the program generates a new one inrelation to the 1D cellular automaton case for each area, and performsself-recovery. After the initial estimation of a pixel-wise disparitymap is enhanced, the program removes holes and artifacts by globallyapplying the self-reproduction of the 2D automaton. Subsequently,backward mapping and non-linear optimization are applied to obtain adense sub-pixel disparity map, in a similar manner to the 1D cellularautomaton case.

FIG. 65 illustrates generation of a genetic pattern according to anembodiment of the disclosure

Specifically, FIG. 65 illustrates an example of generating a geneticpattern through 2D cellular automaton evolution from a genetic supportpattern generated by a 1D cellular automaton.

According to an embodiment of the disclosure, [Equation 33] to [Equation35] for a quantity type-state update function may be applied to apartial sum table having approximated the performance of evolution for a1D or higher-dimensional cellular automaton. An embodiment of theapproximated performance provides dense depth reconstruction by use of agenetic pattern having a line size exceeding one pixel. In fact, for acell size of S_(f)×S_(f) pixels, a sub-pixel disparity map may have alower resolution than an original IR camera image based on a factorS_(f). Further, convolution approximation may be performed to recoverintermediate values between cells. In this case, since the number ofconvolution computations is smaller than in the foregoing case, aninitial estimate may be used to thereby select a set of kernels for eachposition.

Evolutionary algorithms are the names of algorithms for an evolutionarycalculation field that is a lower field of natural computing. Theevolutionary algorithms are a paradigm for solving search andoptimization problems in high-order combinations or continuous searchspaces, inspired by the idea of using the principle of naturalevolution. The most widely known examples are genetic algorithms,genetic programming, evolutionary strategies, and evolutionaryprogramming.

A general operating principle for all cases of evolutionary algorithmsis based on a program loop involving optimization evolution ofsimplified implementation for operator modification, recombination,selection, and a set of candidate solutions (often, called thepopulation of individuals) to given problems. In this generalconfiguration, a modification corresponds to a modification to onecandidate solution which typically prefers fewer modifications to moremodifications. Recombination corresponds to the exchange of componentsbetween two or more candidate solutions. An evolutionary process towardpopulations increasing average suitability is driven by preferringbetter candidate solutions, which will be grown with a higherprobability until the next generation, to poorer candidate solutions.Suitability evaluation is intended to calculate the measurement ofexcellence related to a candidate solution. That is, a suitabilityfunction corresponds to an available object function of an optimizationproblem,

Evolution of parameters of complex algorithms may seem to be a designproblem that, although a reverse design problem, that is, a targetdesign (the behavior of an algorithm to be parameterized) is known, themethod of implementing the target design is not known. A reverse designof a cellular automaton corresponds to the problem. A cellular automatonis used to generate a global state by using local rules in many fields.Finding rules that display an intended aspect may be a difficult work inrelation to the problems of an actual world.

Hereinbelow, evolution of a cellular automaton at epoch t^(i) to epocht^(i+x) according to synchronous update ([Equation 1] and [Equation 2]),asynchronous update ([Equation 15] to [Equation 17]), or approximatedperformance ([Equation 33] and [Equation 35]) is considered.

In a 2D lattice, the cellular automaton has states C_(t) ^(i) in epocht^(i), and states C_(t) ^(i+x) in epoch t^(i+x) a. The size of atransition rule will become larger when possible, according to the limitof power consumption. Therefore, the suitability function of a geneticalgorithm, according to the disclosure, may be introduced as thedifference between C_(t) ^(i+x) and an estimate based on the transitionrule δ^(T). Since the approximation of performance for cellularautomaton evolution is applied, the concept of prediction is used in thecontext of probability. Therefore, the genetic algorithm (GA) requiresan optimum solution or transition rule that provides an accurateapproximation of cellular automaton forward evolution δ_(x)^(T):t^(i)→t^(i+x) or backward (reverse) evolution δ_(x)^(T):t^(i+x)→t^(i) with probability μ.

The GA evolves a population of L transition rules (individuals eachhaving a size of size 9×9 bits (=81 bits)) for detecting what satisfiesthe probability μ or the best performance. It is assumed that the GAuses tournament selection to determine whether to maintain anyindividual alive or not. This involves execution of ‘tournaments’ forthe population to determine the next generation. In each tournament, qindividuals are selected randomly from generation t, and an individualwith the highest suitability is copied for generation t+1. This isrepeated until generation t+1 has the same number of individuals asgeneration t.

After the selection, a recombination is applied to generation t+1. Therecombination is performed by modifying the resulting individuals bymeans of single point crossover for a subset of the population and aprobabilistic bit flip. The relative number of individuals used in thecrossover is denoted by a crossover rate c. The modification isperformed by inverting all bits of individuals with probability m. Forexample, a small crossover rate may be selected for the purpose ofdetecting minimum mini-patterns for additional performanceapproximation. All individuals within the population are initializedrandomly by normalized distribution of the number of 1 s of bit codes ofthe individuals. This implies that the number of individuals having aspecific number of 1 s is roughly equal to the number of individualshaving a different number of 1 s. This prevents an algorithm fromspecializing a specific area of a search space at the beginning of thealgorithm. After the best individuals of D generations and the lastgeneration are best solutions to specific positions of cells (i, j) inthe cellular automaton lattice, the procedure ends. This means that thesuitability function used in the algorithm does not need to beprobabilistic, and all such problems may be combined into one problem.Accordingly, the number L of individuals is selected according to thenumber of cells in a 9×9-cell lattice, and herein, a cell is of sizeS_(f)×S_(f).

The suitability function is evaluated by an average squared errorbetween evolutions based on a transition rule and a cellular automatonrule applied to each cell. Herein, the best solution may be determinedby a softmax function for all optimal solutions of cells.

FIG. 66 illustrates a per-epoch 2D cellular automaton evolution in orderto achieve a quiescent self-organized state according to an embodimentof the disclosure.

FIG. 67 illustrates simplified or approximated performance of a 2Dcellular automaton along with evolution of a global aspect through a GAaccording to an embodiment of the disclosure.

The GA brings a set of optimum transition rules for all cells in a 2Dlattice, and a final quiescent-state genetic pattern is generated bydirectly applying the set of optimum transition rules to an initialstate. That is, the cellular automata rules of [Equation 12] may bereplaced with transition rules in the following form.

$\begin{matrix}{\begin{matrix}a_{{- c},{- c}} & a_{0,{- c}} & a_{c,{- c}} \\a_{{- c},0} & a_{0,0} & a_{c,0} \\a_{{- c},c} & a_{0,c} & a_{c,c}\end{matrix} = {{> \begin{matrix}a_{{{- 3}c},{{- 3}c}} & a_{{{- 2}c},{{- 3}c}} & a_{{- c},{{- 3}c}} & a_{0,{{- 3}c}} & a_{c,{{- 3}c}} & a_{{2c},{{- 3}c}} & a_{{3c},{{- 3}c}} \\a_{{{- 3}c},{{- 2}c}} & a_{{{- 2}c},{{- 2}c}} & a_{{- c},{{- 2}c}} & a_{0,{{- 2}c}} & a_{c,{{- 2}c}} & a_{{2c},{{- 2}c}} & a_{{3c},{{- 2}c}} \\a_{{- c},{- c}} & a_{{- c},{- c}} & a_{{- c},{- c}} & a_{0,{- c}} & a_{c,{- c}} & a_{{2c},{- c}} & a_{{3c},{- c}} \\a_{{{- 3}c},0} & a_{{{- 2}c},0} & a_{{- c},0} & a_{0,0} & a_{c,0} & a_{{2c},0} & a_{{3c},0} \\a_{{{- 3}c},c} & a_{{{- 2}c},c} & a_{{- c},c} & a_{0,c} & a_{c,c} & a_{{2c},c} & a_{{3c},c} \\a_{{{- 3}c},{2c}} & a_{{{- 2}c},{2c}} & a_{{- c},{2c}} & a_{0,{2c}} & a_{c,{2c}} & a_{{2c},{2c}} & a_{{3c},{2c}} \\a_{{{- 3}c},{3c}} & a_{{{- 2}c},{3c}} & a_{{- c},{3c}} & a_{0,{3c}} & a_{c,{3c}} & a_{{2c},{3c}} & a_{{3c},{3c}}\end{matrix}} = \delta_{x_{i,j}}^{T}}} & \left\lbrack {{Equation}\mspace{14mu} 37} \right\rbrack\end{matrix}$

According to an embodiment of the disclosure, transition rules may beapplied, instead of the origin of evolution of a cellular automaton.

Therefore, synchronous update ([Equation 1] and [Equation 2]) andasynchronous update ([Equation 1] to [Equation 3]) for each cell in anautomaton are replaced with [Equation 37]. According to the abovedescription, this method convolutes a captured image of a reflectedpattern with a transition rule around a cell. That is, a partial sumcomponent a_(0,0) of a transition rule matrix ([Equation 37]) iscentered on the position (i, j) of a current cell in the cellularautomaton lattice. Along the cells of the lattice, this method applies atransition rule δ_(x) _(i,j) _(t) _(i+x) _(→t) _(i) ^(T) correspondingto each position and repeats this operation L times, L is introduced asthe number of cells and the number of transition rules. To decode aspatial code from a genetic pattern, two methods for using a GA areavailable. One of the methods is to decode a spatial code by usingforward transition rules for self-reproduction of a cellular automatonin the following synchronous and asynchronous modes,

a _(ij) ^(i+x)=δ_(x) _(i,j) ^(T) *a _(ij) ^(i)   [Equation 38]

or if cellular automata rules are reversible, solve the inversionproblem.

a _(ij) ^(i)=δ_(x) _(i,j) ^(T) *a _(ij) ^(i+x)   [Equation 39]

The genetic pattern approximates transition rules as a kernel andperforms an approximation function of convolving [Equation 38] or[Equation 39] in the same manner as described in regards to theconvolution-based methods and cellular automaton evolution ([Equation34]). Therefore, the genetic pattern for cellular automaton evolutionbased on the strengths of actual values is given as follows.

a _(i,j) ^(i)=δ_(x) _(ij) ^(T) *a _(i,j) ^(i+x)=Σ_(k=0) ^(K) R _(k)^(approx)   [Equation 40]

For the binary genetic pattern as illustrated in FIG. 68,

FIG. 68 illustrates connected components within a transition rule duringapproximation as a convolution kernel according to an embodiment of thedisclosure.

Therefore, a set of transition rules according to the ‘method ofapproximating a multimedia signal filtering algorithm’ may berepresented as a set of kernels in one layer. In addition, sincetransition rules are 2D, they may be combined into one 3D array. Thekernels are permuted in the array, thereby enabling detection of theminimum number of mini-patterns or connected components for thestrengths of actual values and a binary genetic pattern based on thestrengths. In that manner, the method of the disclosure may execute anapproximation function for convolving a genetic pattern and transitionrules in a captured image.

The afore-described encryption using 1D and 2D cellular automataprovides a depth point cloud density corresponding to an IR imageresolution having sub-pixel accuracy. A depth resolution is a 2^(nd)order function of depth. This means that a disparity in one pixel coversanother range of depths, which increases a distance from a camera.

FIG. 69 illustrates the decrease of a point cloud density according tothe disclosure.

Specifically, FIG. 69 illustrates that if a density is always 307200points (=640×480), the point cloud density is decreased according to theincrease of a distance to a camera.

To eliminate this dependency, an IR projector or an IR camera lens mayvibrate with respect to an optical axis by applying a magnetic field.Therefore, the camera may capture various images with views at variousangles. It is important that the angles should be of small and knownvibration orders. Therefore, the program (or the processor) may processa few frames to statistically increase depth accuracy and increase theresolution to a maximum distance along a working area. Therefore, theprogram performs encryption and depth estimation in each frame. Herein,merging of all reconstructed point clouds is applied through rotationconversion corresponding to known lens vibration angles.

The system and method for reconstructing a 3D environment having anapproximated performance asserted by the mobile device (the ‘method ofapproximating a multimedia signal filtering algorithm’) may performsimultaneous local area mapping for 3D content generation, and 6 degreesof freedom (DoF) location tracking and robust body pose estimation of acamera (hereinafter, referred to as, location tracking) as disclosed inthe following ‘Generation of 3D Content’. In this relation, embodimentsof the disclosure may be used to use an extra image of an environment,including subtraction of a genetic pattern in an IR image, andsegmentation of objects along with recognized shadows and a 3Denvironment map. According to a method of generating a 6 DoF userexperience (UX) (hereinafter, referred to as ‘3D content generation’),the last one may be used in area mapping. At this time, key frames maybe stored in a memory at vertexes of a virtual polyhedron having edgescorresponding to a baseline between the camera and the projector.Further, middle point clouds (map) within the volume of the polyhedronmay be reconstructed, and the resolution may be enhanced, through thedisclosed complex view synthesis (hereinafter, referred to as ‘locationtracking’). The ‘method of approximating a multimedia signal filteringalgorithm’ is used for approximation of a convolution neural networkfrom the ‘location tracking’. The approximated performance may be usedin 3D interpolation for sub-pixel location tracking and robust body posetracking for a natural user interface in VR/AR.

FIG. 70 is a flowchart illustrating a method of calculating depthinformation for a 3D image according to an embodiment of the disclosure.

The method of calculating depth information according to the embodimentof the disclosure may include generating a pattern on the basis of thevalue of at least one cell included in a 2D image in operation 3710,projecting the pattern in operation 3720, capturing a reflected image ofthe pattern in operation 3730, and calculating depth information on thebasis of the reflected image of the pattern in operation 3740.

The generation of a pattern may include setting a value for each of aplurality of cells included in the 2D image. For example, the value setfor each of the plurality of cells may be 0 or 1.

Further, the value of the at least one cell included in the 2D image maybe an initial value set for a row or column of a plurality of cellsincluded in the 2D image.

The generation of a pattern may include determining, on the basis of thevalue of one cell included in the 2D image and the values of two cellsneighboring the one cell, the value of another cell neighboring the onecell.

Specifically, the generation of a pattern may include determining, onthe basis of the value of the one cell included in the 2D image and thesum of the values of the two cells neighboring the one cell, the valueof another cell neighboring the one cell.

In this case, the calculation of depth information may includegenerating a pattern on the basis of the values of cells in a row orcolumn of a plurality of cells included in the reflected image of thepattern.

Further, the generation of a pattern on the basis of the values of cellsin a row or column may include determining, on the basis of the value ofone cell included in the row or column and the values of two cellsneighboring the one cell, the value of another cell neighboring the onecell,

Herein, the calculation of depth information may include comparing thevalues of a plurality of cells included in the reflected image of thepattern with the values of a plurality of cells included in the patterngenerated on the basis of the values of the cells in the row or column.For example, the depth information may be determined on the basis of aresult of the comparison.

Meanwhile, the method of calculating depth information, according tovarious embodiments of the disclosure, may be implemented as a programcode executable by a computer, and provided to each server or devices sothat the program code may be executed by a processor in the state whereit is stored in any of various non-transitory computer-readable media.

For example, a non-transitory readable medium storing a program forperforming the operations of generating a pattern on the basis of thevalue of at least one cell included in a 2D image, projecting thepattern, capturing a reflected image of the pattern, and calculatingdepth information on the basis of the reflected image of the pattern maybe provided.

The afore-described various applications or programs may be stored andprovided in a non-transitory readable medium such as a compact disk(CD), a digital versatile disk (DVD), a hard disk, a Blue-ray disk, auniversal serial bus (USB), a memory card, and a read only memory (ROM).

As is apparent from the foregoing description, according to thedisclosure, a 3D environment can be modeled by calculating depthinformation.

While the disclosure has been shown and described with reference tocertain embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

Although the present disclosure has been described with variousembodiments, various changes and modifications may be suggested to oneskilled in the art. It is intended that the present disclosure encompasssuch changes and modifications as fall within the scope of the appendedclaims.

What is claimed is:
 1. A method of obtaining depth information, themethod comprising: obtaining a pattern based on a value of at least onecell included in a two-dimensional (2D) image; projecting the pattern;capturing a reflected image of the pattern; and obtaining depthinformation based on the reflected image of the pattern
 2. The method ofclaim 1, wherein obtaining the pattern comprises setting a value foreach of a plurality of cells included in the 2D image.
 3. The method ofclaim 2, wherein the value for each of the plurality of cells isselected from integers equal to or larger than zero.
 4. The method ofclaim 1, wherein the value of the at least one cell included in the 2Dimage is an initial value set for a row or column of a plurality ofcells included in the 2D image.
 5. The method of claim 1, whereinobtaining the pattern comprises determining, based on the value of theat least one cell included in the 2D image and values of two cellsneighboring to the one cell, a value of another cell neighboring to theone cell.
 6. The method of claim 5, wherein obtaining the patterncomprises determining, based on the value of the at least one cellincluded in the 2D image and a sum of the values of the two cellsneighboring to the one cell, the value of another cell neighboring tothe one cell.
 7. The method of claim 1, wherein obtaining depthinformation comprises generating a pattern based on values of cells, ina row or column, of a plurality of cells included in the reflected imageof the pattern.
 8. The method of claim 7, wherein obtaining the patternbased on the values of cells in a row or column comprises determining,based on the value of one cell included in the row or column and valuesof two cells neighboring to the one cell, a value of another cellneighboring to the one cell.
 9. The method of claim 7, wherein obtainingdepth information comprises comparing values of a plurality of cellsincluded in the reflected image of the pattern with the values of aplurality of cells included in the pattern generated based on the valuesof the cells in the row or column.
 10. The method of claim 9, whereinthe depth information is determined based on a result of the comparison.11. A terminal comprising: a projector; a capturing unit; and aprocessor configured to: obtain a pattern based on a value of at leastone cell included in a two-dimensional (2D) image, control the projectorto project the pattern, control the capturing unit to capture areflected image of the pattern, and obtain depth information based onthe reflected image of the pattern.
 12. The terminal of claim 11,wherein the processor is configured to obtain the pattern by setting avalue for each of a plurality of cells included in the 2D image.
 13. Theterminal of claim 12, wherein the value for each of the plurality ofcells is selected from integers equal to or larger than zero.
 14. Theterminal of claim 11, wherein the value of the at least one cellincluded in the 2D image is an initial value set for a row or column ofa plurality of cells included in the 2D image.
 15. The terminal of claim11, wherein the processor is configured to obtain the pattern bydetermining, based on the value of the at least one cell included in the2D image and values of two cells neighboring to the one cell, a value ofanother cell neighboring to the one cell.
 16. The terminal of claim 15,wherein the processor is configured to obtain the pattern bydetermining, based on the value of the at least one cell included in the2D image and a sum of the values of the two cells neighboring to the onecell, the value of another cell neighboring to the one cell.
 17. Theterminal of claim 11, wherein the processor is configured to obtain apattern based on values of cells, in a row or column, of a plurality ofcells included in the reflected image of the pattern.
 18. The terminalof claim 17, wherein the processor is configured to determine, based onthe value of one cell included in the row or column and values of twocells neighboring to the one cell, a value of another cell neighboringto the one cell.
 19. The terminal of claim 17, wherein the processor isconfigured to compare values of a plurality of cells included in thereflected image of the pattern with the values of a plurality of cellsincluded in the pattern generated based on the values of the cells inthe row or column.
 20. The terminal of claim 19, wherein the depthinformation is determined based on a result of the comparison.